I have waited awhile to chuck my tuppence into the ‘Climategate’ ring as I am not a true believer and have preferred to see how things played out over time. I personally favour the hypothesis that humans are causing some climatic effects but I do not believe the evidence is sufficient to prove my opinion correct. I have an open mind towards those who are weakly opposed, which is to say those who are waiting on data to prove me wrong. If one were to place my opinions on a dartboard, I would fall pretty close to the one labeled Bjorn Lomborg.
I am far more worried about the collateral damage the CRU researchers have caused. Their machinations, exposed by these ‘Pentagon Papers’ of the oughties, is damning and damaging to public trust in science. It opens the door to all sorts of pseudo-science by making their expositors appear superficially to be as trustworthy as the real thing. This is bad. This is very bad.
‘Climategate’ is not the first case of serious scientific fraud in recent years but may be the most damaging and far reaching one. Other well known cases included South Korean scientist Hwang Woo-Suk who falsified his work on cloning and Bell Labs Physicist Jan Hendrik Schön who faked results in numerous papers. Schön used the same fake graph, with modified labels, in three totally different papers. That was just a starter. His massive misdeeds caused a gravity wave ripple through the Physics world as every paper citing his work had to be reconsidered.
Since all CRU citations must now be treated as problematic, the potential of ‘Climategate’ is not a ripple but a terrible and destructive tsunami. The researchers responsible for this have set their field back by years and should be disciplined by their peers accordingly. Cleaning primate cages is too good for them.
A second facet of ‘Climategate’ is the reported shortcomings in the model code base. Part of the document release included source code. In a discussion with Rand Simberg over breakfast in LA earlier this month I heard that some very knowledgeable open source programmers are having a go at it. If half of what he told me turns out to be true, the models used by IPCC are worse than useless.
I have several times in my career translated serious numerical modeling code from Fortran to modern languages and thus had to deal with the issues of validating the results. In the real world mistakes cost money and sometimes lives. Most recently I translated some aerodynamics code for a New Space company. I spent weeks doing nothing but validating and checking to be sure the output was reasonably trustworthy for questions within the realm of interest. When Rand told me the CRU model code did not even handle numeric overflows I was speechless.
Let me explain. Computers represent numbers in binary. Any signed representation (ie one that handles plus and minus) will use some formatting trick to differentiate the two. The problem is, if a positive number gets incremented to be one bit too big… it may suddenly become a negative number. Regardless of what does happen, any calculation using the value after an overflow might as well be a random number generator. The results are totally, utterly worthless. There is not a chance in hell that the output will be meaningful.
There are ways of dealing with this sort of thing but I will not go into that sort of techno-detail here. My goal is simply to point out that if the statements I heard are true, I must cease to believe the validity of any output from CRU and CRU related models.
There is really only one acceptable way for the field to recover credibility and reinvigorate trust. The code for models must all be made open source. It must be released into the public domain where experts in numerical programming can openly argue about the validity of the code, the mathematical techniques and the mathematical and physical simplifications and assumptions it contains.
I will no longer believe results which lack this corroboration. If an author refuses, I am going to assume they have misdeeds to hide.
Early in this article I said I lean towards pro on the hypothesis of human caused climate change. I should expound upon that a bit more. It is my belief that we are causing some change at present and if things went on as they are now there might be some serious, but not civilization threatening results.
However, things are not going to stay the same. A collapse in carbon output is going to occur and the reasons for it have nothing to do with cap and trade or Copenhagen or any other State or NGO foisted crisis plan. By the middle of this century liquid fuels such as gasoline will be generated using the Fischer-Tropsch process in some updated form. It will be carbon neutral because part of the feedstock will be free for the taking: atmospheric CO2. It will be split using either grid power, mechanical nanotechnology or genetically modified algae (some of which is purportedly working already). With the addition of energy, CO2 -> CO + O, and the Carbon Monoxide may be fed into the same FT process that was used to fuel the Nazi war machine. Towards the end of World War II this was nearly the only source of fuel available to Germany. Anyone who believes this technology is unproven on an industrial scale is simply historically ignorant.
Carbon based grid power is already declining as a relative portion of US energy (30% according to a recent SciAm article). I expect that decline to accelerate as use of ever cheapening and ever improving solar panels really starts to bite. We will also see inputs from Space Based Solar Power growing explosively by 2050. New technology nuclear and perhaps even game changing wild cards like Polywell Fusion will be taking up major roles by then as well.
If you toss in the huge impacts nanotechnology will have on all facets of technological civilization and the expected population decline in the second half of the century one begins to wonder exactly what will be the climate change problem of 2100? If human CO2 inputs collapse and population declines what climatic impact will the modeling of that scenario show?
There is yet another wildcard to consider. What if we are about to hit a Maunder type solar minimum? There is debate on this issue but it is certainly not closed. Such a decline could cover any human global warming until long after we have transitioned to more modern energy sources.
We need Climate Science to cleanse itself of political hacks. Young scientists must learn that Science cannot save Politics… but Politics can certainly ruin Science. Let scientists generate science and leave politicians to deal in the realm of opinion and ‘what people want’.
Good thoughts here.
An open and alert mind, how unusual.
Especially note the comments on bogus climate model code (this seems to be what Freeman Dyson objected to. I think he used the word, fraud), and the idea that the inputs to our energy economy are bound to change almost beyond recognition and soon.
Copenhagen and the AlGore Global Warming Scam are far more interested in political power and divying money than in the climate.
Dale, I’ve been following climategate very closely since the story broke, and yours is by far the most intelligent contribution that I’ve seen from either side.
Well done.
Bravo – one of the best posts I have read about Climategate.
The trouble is the politicisation and irrationality go very very deep in the science. I have had some alarming conversations with friends of my children who are science undergrads this christmas to paraphrase “So you’re telling me that you dont care how manipulated indeed false the UEA material might be……..we still need to deindustrialise now for scientific reasons?!”
All of them are getting your post.
Good post Dale.
What I fail to grasp is how so many people were taken in by this nonsense. Any person with a rudimentary high-school science education could easily see through it, even before the climategate papers.
One could see that hockey-stick claims are ridiculous, “tipping points” totally unsupported by any scientific argument, that Venus-like scenarios are just foolish propaganda.
I’m at a loss to comprehend how people were taken in by such rubbish. Maybe they don’t teach any science nowadays in high school, and people in the humanities, which is the great majority, are totally ignorant of the physical world.
The sad thing, Dale, is that we are not seeing an appropriate reaction from “official” science. How many of the scientific organizations that put out politically correct statements warning of alleged anthropogenic global warming over recent years have now publicly suspended those statements pending a re-evaluation of the supposed scientific basis? None that I am aware of.
The driver for alleged anthropogenic global warming (renamed “climate change” when the globe refused to warm) has been government grants; billions of dollars of government grants. And that driver still remains.
Too many “scientists” have chosen to become whores. Until someone offers then a better deal, they will continue whoring for alleged anthropogenic global warming. Official science may never recover its credibility.
Excellent post. The “hundreds of lines of uncommented FORTRAN” email should be your favourite — that blew my socks off. I was amazed that the raw data and source code was not fully validated and available to all for verification. It is simply astounding.
My own tuppence is at assassinationscience.com/climategate.
It’s not just climate science that’s discredited. Shouldn’t we now go back and look at second hand smoke, effects of low-dosage alcohol on driving and every other scientific study in aid of ‘trendy’ causes? The twin corrupting influences of grants and social approbation may have been in play there, too.
Nikola Tesla was a scientist, Benton Quest was a scientist. The grant-whoring political hacks at the CRU are not scientists.
I miss living in a world run by grown-ups.
The love of [grant] money is the root of all evil.
Every time a haughty PhD refuses to answer questions from the non-scientific community about a theory and instead insults their intelligence, or insists on calling theory ‘fact’ or ‘proven’ when it is not, it damages public trust in the scientific community.
What if other scientists call B.S. on their fanatical or prostituted brothers who rain insults on the curious public who have questions? What if scientists offered clear explanations? What if they weren’t so hostile and defensive with non-scientists, or even religious people? What if they really used the scientific method and started teaching it on YouTube & TED? It certainly isn’t being taught in public school or many universities.
Start there and maybe you can restore public trust.
Dale
While I’ve always respected Lomborg (Not to mention you) I’m still not sure why you buy into any aspect of the AGW theory. Sure there might be a minimal effect on climate from human activity, The effects of Cloroflorocarbons on the Ozone layer seemed to me pretty well established, though I’m ready to have my mind changed. The Earth’s climate is overall such a huge, complex chaotic system that reducing it to any single element is ridiculous.
I note that the left is still calling CO2 a pollutant.
Anyway , congrats on an excellent post.
Alice:
The prevailing scientific view changes slowly and cautiously, and rightly so. But this statement by the American Physical Society suggests that, if warranted, they may eventually be taking the kind of step you envision.
@John Costella – Your link is welcome, but the page seems to cut off abruptly in the middle of March 12 2003.
Dale, thank you for your considered post.
All at Samizdata, Best wishes for 2010 and keep up the exceptional work.
Taylor,
Even the chloroflourocarbon – ozone depletion theory is now in trouble. One of the intermediate reactions has been shown to run an order of magnitude too slow under the condition that exist in the atmosphere where this happens. According to one of Jerry Pournelle’s (www.jerrypournelle.com) correspondents a couple of years ago there was a large NASA project to measure the intermediate reaction products and they only found 5% of the predicted levels so the results were buried.
As for CO2 it certainly isn’t a pollutant and any effects on global temperatures are currently buried in noise both in the scientific and political sense. Until the CRU , GISS and NCDC data is examined and verified by independent “red teams” the null hypothesis must now be that it is useless.
positive number gets incremented to be one bit too big… it may suddenly become a negative number
Not ieee floats, they go to inf; they are signed numbers as opposed to two’s complement. The problem with the code in question was that it was squaring *integers* before accumulating the result in a float. I guessed that without even looking at the code and it turned out to be the case. I was surprised that the programmer who noted the problem couldn’t locate and fix the root cause.
“May” have been?
I don’t know about every study in aid of ‘trendy’ causes, but the classic second hand smoke study was notoriously fraudulent:
http://www.numberwatch.co.uk/2003%20April.htm
And as for low-dosage alcohol, The Science Is Settled (TM): drivers who have had one drink are actually safer than those who have had none:
http://www.numberwatch.co.uk/2008%20April.htm
Oh what, They didn’t tell you these facts? Gee, I guess They must have forgotten to send out the press releases. I mean, it’s not like They are a corrupt tax-funded oligarchy or anything like that..?
Tar. Feathers.
I got as far as that and gave up. To characterise this trivial affair as ‘the most serious scientific fraud in recent years’ is so far off the mark you clearly have no idea what actually happened, nor its relevance to reality.
The reality is that nothing of any substance against the AGW theory has emerged from those emails. Even if you ignore all the CRU work, it changes nothing at all. Virtually every other piece of independent data and work supports the broad conclusions of CRU.
Sorry, nice political puff piece to rally the troops with, but that is all.
“Let scientists generate science and leave politicians to deal in the realm of opinion and ‘what people want’.”
I am perfectly happy for scientists to extend the reach of reason ever further into the physical world, but I will not “let” politicians deal with “the realm of opinions” and “what people want” because that is naturally the concern of free individuals making decisions in free markets. Moreover, I will neither thank, trust nor defend anyone else who is happy to “let” any of the values I act for in any market to fall into the death pit of politics.
Babba, it’s always nice to be reassured by the voice of authority. You are an authority on climate science, aren’t you?
Great piece, but like others I’m puzzled as to why an avowed sceptic (in the admirable sense of the word) would continue to buy into the CO2/AGW catastrophe myth at any level, even one as highly nuanced – and probably lonely – as yours. Surely the first logical conclusion to be drawn from the available science, once it has been relieved of all now-discredited literature (a much larger body of work than Babba seems to imagine), is that no real evidence exists that there is anything whatsoever to be scared of?
For a great analysis of why part of every generation seems to believe it is going to be the last unless (insert catastrophe), and of the generally woeful results of government “response” to such scares, see
http://kestencgreen.com/green%26armstrong-agw-analogies.pdf
Excellent post.
Babba, just like every defender of the CRU I have seen who decides to mention the scandal, focuses on the emails and ignores the code completely. The code is much more damning, and essentially proves that anything citing the CRU’s model is complete bollocks. As this was the biggest contributor to the IPCC’s reports as well as the prime reason given for the EPA declaring CO2 a pollutant, i’m quite prepared to call the whole thing bollocks.
I think part of the appeal to many people of AGW is that it says “Look at humanity, we’re powerful enough to change our world’s climate.” The green’s corollary to this is “and you must all be punished for it.” Note the “you” – it is most emphatically not a “we”.
This post starts by comparing the CRU emails with people who falsified work and others who faked results.
Next based on this daming comparision Dale Amon asserts “all CRU citations must now be treated as problematic”
Based on what? What is lacking here is presentation of the damning evidence in its proper context. I’ve repeatedly asked commenters making similar claims for substantiated evidence that is more than speculation and innuendo (about what out portions of email might mean). Responses have ranged from more bluster to non-responses.
How about you present the evidence on which you have found them guilty of falsifying results?
I think part of the appeal to many people of AGW is that it says “Look at humanity, we’re powerful enough to change our world’s climate.” The green’s corollary to this is “and you must all be punished for it.” Note the “you” – it is most emphatically not a “we”.
Well said, Mart.
And a very interesting post, Dale, indeed. To lay the whole thing open to full review by anyone is indeed the only way to restore any credibility.
As soon as people get secretive the only appropriate response is suspicion.
A thought: What will happen when the energy absorbed as electrical energy by solar panels is happening on a global scale, similar in scale to that now of current energy production and use? And more so if captured in space and delivered for use on the planet.
Will we be capturing energy that otherwise would have been reflected and thereby cause unintended consequences?
Jakeman
CRU’s own programmers described their data as far worse than ‘problematic.’
Plus gaming the peer review process, thwarting FOI requests, admission of replacing proxy data with instrument data when the proxy temps fell.
It’s interesting that you ask skeptics to ‘present the evidence’ but this entire scandal revolves around the refusal of Jones et al to present the raw data on which they make their claims.
Fortunately that seems about to change.
How about you present the evidence on which you have found them guilty of falsifying results?
Hide The Decline
Excellent post, very clear disposition of events.
Good point about population decreasing – the Economist had stated by 2050 the Earth’s population will begin to decrease as modernization takes hold.
I am not impressed by the apocalyptic talk of the Greenies – they are never held to account for their claims. Check the Earth Day predictions of 1970 – they were promising deaths of billions from pollution, famines, resource depletion, etc. and none of it, absolutely none of it is true. Their modus operandi is too yell fire in a crowded theater because they are a protected class of citizenry who are immune from punishment and will never be held to account, at least by the press, for blatant and irresponsible falsehoods. This has led to an arrogance that permeates the entire CRU attitude towards the public. They are free to lie since they are the noble ones saving the Earth. Let’s stop that now.
You are absolutely right about collateral damage. Many honest, careful climatologists have innocently relied on the fraud coming out of Hadley/CRU and NASA/GISS. Their own work is now contaminated, and it will take years to undo the damage. Whole careers have been wasted.
Worse, a entire, valid scientific discipline and all its practitioners are now in disrepute. All of it, everyone. Intelligent graduate students will begin to avoid the discipline just when they are most needed. The hacks will take over.
It is because of such articles as this I’ve come to value this place ever more so. Thanks.
The only solution available for anyone making extraordinary claims is to provide all the data. An informed choice is impossible absent the full substance of evidence.
It certainly isn’t unreasonable to think, that absent the full disclosure by those who demand political solutions, that their motives are less than honorable if they withhold data.
Jakerman:
For any scientific claim to be treated as possibly valid, all i’ts data and calculations (code) must be placed in the public domain. Independent scientist must replicate the results.
No scientific claim can be considered valid until the replication has been done several times over.
All CRU citations should never have been treated as anything but unsubstantiated hypotheses. It’s not the climategate e-mails or code that proves them invalid. It’s the fact that they have never been checked and replicated by independent scientists. It is the fact that CRU “scientists” (and other climate scientists) have, knowingly and intentionally, hidded their data and code so that no one would be able to check their claims. It’s something we knew all along, the cimategate e-mails only added explicit acknowledging of this hiding by the CRU “scientists”.
As far as I am concerned, the emails show that some of the individuals who sent or are mentioned in them are not to be trusted.
The code proves that their model and anything based on its results cannot be trusted.
The former damages the reputation of scientists the world over, while the latter is an axe blow at the base of the AGW tree. Will it withstand this blow? I’m not so sure, as it seems to have revealed a lot of hidden rot.
There simply isn’t enough data to measure global temperatures with the precision claimed. Errors multiply, with every step of the calculation, but not enough has been done to keep track of them, so the final error bars are much too narrow.
I suspect, but can’t prove from my armchair, that the field also suffers from circular reasoning. People are calibrating various ways of measuring temperature against each other, but none of them has a fixed reference point, so the whole lot can drift out of touch with reality, because ultimately they’re being calibrated against themselves. The researchers may have thought they had a fixed point, the CRU data, but that includes adjustments intended to make the results match other researchers.
This kind of circular reasoning can happen easily when methods are concealed, as here, which is one reason for openess.
Very cogent article, Dale.
From my perspective, the worst aspect of the entire state of human reaction to the potential of significant planetary climactic shift is that the currently politically favorable ‘solution’ appears to quite likely place human beings in an unfavorable posture to effectively cope with it.
At the crux is quite simply the management of energy. While the underlying motivation of the greens might be quite admirable, i.e. pressuring the species to make much more economical and efficient uses of energy and the transfer mechanisms involved, their methodology is utterly deplorable. The use of scare tactics and what on an almost daily basis increasingly appear to be equivocation and statistical chicanery of dubious basis and validity may very well have to ultimate effect of damaging progress towards an ultimately admirable goal and desired outcome. That outcome being an enhanced human ability to deal with climate changes in a manner beneficial to humans as a whole.
The subject is currently under the thrall of political opportunists working diligently for the short term and personal benefits of money and power (Gore, Pachauri, et.al.); Academics intoxicated with attention, status, and public funding (Mann, Hansen, Biffra, Schmidt, et.al); legions of scientifically ignorant agenda journalists (Borenstein, et. al.) – all of whom have made common cause of overlapping interests to foist an almost ridiculously fantastic ‘chicken little’ scenario upon a self absorbed and too ready to be guilt-ridden ADD afflicted public.
Consequently, in some respect, we’re not in a very different position than that of villagers convinced that the only way to ensure a good harvest for the next year is to toss one of the village virgins into the nearest volcano.
The sooner and more completely these frauds and charlatans are exposed, the better – as there is still a massive amount of real science necessary to come to grips with what’s actually happening with the dynamics of our little corner of the Milky Way galaxy.
Which will only get underway once the screaming and wailing about the boogeyman under the bed is dealt with.
Dale,
As someone who loved working in Belfast with the folks in at the Queens University Physics and Mechanical engineering departments (I am guessing that you were there before moving to Wyoming) your post was a breath of fresh air.
You are looking ahead to what will happen when the fossil fuels are exhausted. My guess is that new nuclear reactors will be needed and especially ones that can consume nuclear waste from current nukes. Candidates include the Liquid Flouride Thorium Reactor (LFTR) and sub-critical reactors based on scale ups of the Spallation Neutron Source (SNS) at ORNL. Known reserves of Thorium should provide the power we need for at least 10,000 years which should be time enough to develop fusion based power generation.
Peter: Actually I am still here. Our Wyoming venture is still very much a distributed corporation until such time as we raise serious capital. That might be this year or it might not. Even then, I will most likely retain my flat here.
I worked for Prof Perrott in the early 90’s btw.
Excellent post, Dale, and not just saying that because your perspective falls pretty close to my own. 😉 Most especially, I can think of no honest reason not to open-source the code for any computer-generated model as a condition of anyone’s ascribing scientific credibility to it. On anything.
I do think human civilization may be just a bit more fragile than you seem to – principally because of the cascading possibilities of human stupidity – and I should be delighted to see your technological predictions come true. I suspect that solar power satellites are going to be mired in political grief until about the twelfth of Never, but I see that in my capacity as SF scribbler alone there are a couple of other matters I ought to be investigating toot sweet.
Still, after all is said and done, the decade has closed without a Fifty-Year Plan for Everything Everywhere; there is a major opportunity to let air and light permanently into some fungoid and niffy areas of climate science; and the dreadful Disaster Environmentalist wing of the Green movement has been knocked unexpectedly onto the back foot.
Yes, I too have hope – and am always happy to be offered further cogent reasons for it.
Cheers!
“I am far more worried about the collateral damage the CRU researchers have caused. Their machinations, exposed by these ‘Pentagon Papers’ of the oughties, is damning and damaging to public trust in science. It opens the door to all sorts of pseudo-science by making their expositors appear superficially to be as trustworthy as the real thing. This is bad. This is very bad.”
Err, no.
Which is worse, a) to know there are cockroaches in the kitchen, or b) not to know there are cockroaches in the kitchen ?
Wrong. The answer is not to have cockroaches in the first place because you’ll encourage other vermin. If you do see them, spray well and get rid of them.
This article in Wired (Link)magazine inadvertently shows up what went wrong in CRU.
@David Bishop: Try refreshing the page — you may have loaded it when it was being updated. Cheers, John
Still no charge supporting evidence, the best attempt so far was “Hide the Decline”.
Afraid that doesn’t cut it. http://greenfyre.wordpress.com/2009/11/21/mikes-nature-trick-to-hide-the-decline/
So would anyone else like to prensent the evidence to support the charge that Dale has found CRU guilty of?
I note the only defender of the CRU conspirators “Baba”; who demonstrates so admirably just how dangerous these guys, his post being: viciously insulting of the opposition, ignorant and more convinced of the righteousness of this particular belief system than those drinking the koolaid at Jonestown.
Thank you for excellent article Dale. jakerman you don’t seem to understand that science puts the burden of proof on those making the hypothesis. CRU has hidden their data, so there is no “proof”.
The emails and data file reveal that CRU scientists were trying to deflect questions rather than doing science.
Coniston, I’m asking for evidence to support the implicit charge that the CRU emails show fabrication of evidence, as implied by Dale’s likening them to Hwang Woo-Suk and Jan Hendrik Schön.
Failing that, I’ suggest Dale restating his charge, if it is the case that he is not pronouncing them guilty of fraud.
When you say CRU have “hidden their data” what are you talking about? Can you provide evidence you rely on and give a credible interpretation of what the evidence means?
http://www.skepticalscience.com/What-do-the-hacked-CRU-emails-tell-us.html
Whether it is fraud or not is not the point; the point is that the program they were using is a badly written piece of crap liable to output inaccurate results.
Following on from that, Dale suggests that data and program code should be made open-source, if this happened, the data could be checked against other sources, and thousands of skilled programmers, whether pro-AGW, anti-AGW or somewhere in between, could evaluate and, if necessary, improve the code. If this was done, everyone could trust the results, if scientists refuse to do this, it looks like they’re hiding something. Like if someone won an Olympic medal then refused to take a drug test, people would be suspicious.
Excuse me? Fraud is about the most serious accusation one can make against a scientist and not a word to be thrown about lightly. I notice that not a scrap of actually poor code has been listed here, instead we have Dale passing on what he heard over breakfast with another blogger who heard that some other [un-named] programmers had looked at the code and found flaws. Given the provenance was a stolen email, we have no way of knowing if it was real code, test code, developer’s ‘what-if’ code, or if it ever got used in a published paper. Before using the ‘f’ word you would have to demonstrate that someone knowingly published results they knew to be wrong. You’re a million miles from that.
Incidentally the IPCC uses other data and models in addition to CRU, for example from the NASA Goddard Institute. The NASA data and source code is 100% open source and agrees well with the CRU results.
The ‘Hide the Decline’ thing was less about the data so much as the public presentation, but it does have implications for the validity of climate science. The chain of reasoning is extended and therefore difficult to do as a soundbite.
Part of the claimed evidence relies upon the current warming being larger than natural variation, which therefore relies on our knowledge of long-term natural variation, which relies on paleoclimate reconstructions, which rely on trees acting as good thermometers, capable of measuring annual average temperature to an accuracy of fractions of a degree.
That is in itself a remarkable claim to make, that surely requires a remarkable strength of evidence to back it up. But people seem to pass over it without noticing.
The reliability of the reconstructions depends on the relationship between temperature and tree ring growth being well-understood. Are they actually related? Well, over the early part of the 20th century, the temperature went up, and the tree rings widened. So wider tree rings can be used to deduce higher temperatures, right?
Trouble is, after about 1960, the temperature goes up, but the tree ring widths go down. That would appear to falsify the hypothesis. You can’t tell the temperature from tree rings. They may be related, but not strongly enough to measure even whether the temperature is increasing or decreasing.
But wedded to their hypothesis, they’re not ready to give up yet, and climate scientists have used what is known as “special pleading”. The relationship clearly stops after 1960, but perhaps it worked up to then, and then the rules changed? Maybe it’s something to do with man, and the data after 1960 is polluted by an extraneous effect. If so, you can safely ignore it, and use all the data prior to that point.
Maybe so, but they need to show the data, discuss the problem, and show the evidence that allows them to draw such a conclusion. The problem is that so far as I know, nobody has ever done so.
And when they showed the “evidence” of global warming in the IPCC presentations to the politicians and public, rather than bring this tricky subject up, they simply chopped off the last 40 years of data, hid the endpoint under a spaghetti of other lines, so you would have to look very carefully to notice anything odd.
In the case being discussed here, the data was not only truncated, but smoothed. Smoothing algorithms have difficulties around the end points, and are very sensitive to small details (and hence very inaccurate) at the ends. The problem was, if they used the real data, the tree ring divergence problem dragged the graph down even more dramatically – precisely the wrong message – and cutting off the inaccurate section eliminated the message entirely. So what they did was to chop off the divergent data, paste in the (heavily adjusted) temperature data, smooth, and then cut off the temperature data again. However, because of the way smoothing works, the end of the series was still heavily affected by it, tilting up instead of down. Thus avoiding the need for any inconvenient explanations, and giving the impression of simplicity and hence reliability.
We sceptics already knew they had done this. Steve McIntyre, one of the “thousands” of scientists who officially reviewed the IPCC reports (it’s another lie – there weren’t thousands, and it wasn’t peer review in the sense that reviewers’ comments could be ignored) spotted it and raised a comment. It was rejected by Briffa without explanation. The climategate emails simply added the motive.
Personally, I think a stronger case can be made for the Wang affair being evidence of scientific fraud, but there’s plenty of other meat. But mostly, it’s not so much evidence of fraud in the climategate emails as of poor quality and lack of checking that degrades all confidence in the evidence. People keep saying there are many other lines of evidence, but the problem is they’re all compromised in this way. (They’re still resisting releasing the data and emails for most other areas, though.) They’re all interdependent. But the people saying it mostly take the word of the climate scientists themselves that such evidence exists, and that’s the end of it. In my experience, they rarely know what it is themselves – most don’t even know how the greenhouse effect works. It’s just an excuse for the politics to them.
In any case, by far the worst blow to be struck against climate science is by these scientists themselves. Our confidence in science is based on it surviving the strongest criticism we can devise. Their attempts to shield their hypothesis from criticism by withholding data from examination and hiding contrary evidence sabotages that at root. Only sceptics can validate climate science – that’s fundamental to the scientific method. They’re needed.
And the fact that these so-called scientists neither understood this nor had the confidence to allow their workings to be examined is a damning endightment of the whole area, and of every other scientist who stood by and allowed this to happen unchallenged.
Phil Clarke,
There’s no need to publish code here, since it’s all public domain and widely discussed. But just for you, here’s the comment that led to this code being investigated.
“17. Inserted debug statements into anomdtb.f90, discovered that
a sum-of-squared variable is becoming very, very negative! Key
output from the debug statements:
(..)
forrtl: error (75): floating point exception
IOT trap (core dumped)
..so the data value is unbfeasibly large, but why does the
sum-of-squares parameter OpTotSq go negative?!!”
That’s the comment of the CRU’s own programmer.
I think you may be misunderstanding this burden of evidence thing. it’s not up to us to prove it wrong, it’s up to CRU to prove it right. They have up to now relied upon hiding the code and saying “trust us, we’re experts” to dodge this. But we’ve got the code now (if it isn’t genuine, then prove us wrong by publishing the real code) and it’s rubbish, and they’re not experts, and now we want to see this evidence you keep on telling us is “overwhelming” but which not even the Freedom of Information Act can induce you to reveal. Phil said he would rather delete the file than let anyone sceptical examine it.
Oh, yes, and NASA’s code is no better. We got that earlier, after reverse-engineering from the data revealed multiple bugs, and NASA was forced to reveal it. That took another long struggle. Look for the posts before and after this one.
http://climateaudit.org/2007/09/08/hansen-frees-the-code/
Phil Clarke writes:
This is a good point, but …
And the but is a big one. If the various bits of criticised code were not relevant to any results from CRU, surely by now, CRU would have declared (and demonstrated by publishing their input data, source code and output), beyond reasonable doubt, that lack of relevance. I’m missing that declaration.
Secondly, Phil Clarke writes:
In this, he is not alone. [If pushed, I’ll find links.]
However, if other founts of knowledge are consistent with the output of buggy code (as analysed here and elsewhere), what does that say about the quality of the output of those other founts of knowledge?
I’ll give two views: (i) those other founts of knowledge are of similar quality of that from the deficient processing by CRU; (ii) we are deficient in knowledge of this ‘consistency’.
And if we are deficient in knowledge of consistency, do we really know the relevance (if any, and I have strong doubts about it) of average temperature of the planet?
Best regards
Still no evidence of the fraud implied by Dale.
I suggest that Dale should now clearly state the crimes he believes CRU are guilty of.
jakerman, you seem confused. Dale sent out his position quite clearly. This is a blog, not a forum where people feel the need to fire off a retort every different opinion. Dale has already explianed why he thinks what he does. Several others have also added support, with reasoning, why CRU is hopelessly compromised and why the math is meaningless. You don’t have to agree or like it, but then no one is stopping you explaining why you think Dale is wrong on your own blog and linking to his article that you are criticising.
Perry writes: “jakerman, you seem confused. Dale sent out his position quite clearly”
I’d be glad to read where Dale clearly states what the CRU are guilty of. Can you direct me to where he state this?
Perry writes: “Dale has already explianed why he thinks what he does”.
Again please show me where Dale or anyone supports with evidence his comparison of the CRU with the fraud of Hwang Woo-Suk and Jan Hendrik Schön.
Please show me how Dale or anyone justfies his claim of
“serious scientific fraud”.
I’ve seen plenty of evidence of readers of this blog uncritically accepting Dales assertions of serious fraud without evidence and thanking him for his post. But not anywhere were he support this charge.
I’ll be happy to address other charge you make of the CRU (including your assertions of hopeless and “meaninings”) when this fundamental issue is dealt with. But before that, the charge of serious scientific fraud has bee asserted, yet not backed up.
If certain individuals claim to be scientists (to wit: people who hold to the necessity of rational criticism in the advancement of knowledge), yet knowingly act to shield their work from rational criticism by others, then such individuals are fraudulently claiming to be scientists.
Pieces of paper and letters after your name are professional accolades; they are not a certification against moral failures that may undermine an individual’s membership of that profession itself.
Jakerman,
The charge is supported elsewhere. Dale is simply commenting on what is already well known to most people here. The purpose of this post was not to conduct or even reprieve the trial again, but to comment on its implications and the road to recovery for climate science.
We do care about the fact that there are many people who still believe strongly, and still seek to defend the behaviour, but that’s not a quick or easy job, and others are better equipped to do it. I hope you’ll pardon us if we don’t waste too much time on it.
If you really want a specific example of scientific fraud, then I’ve already alluded to the Wang case. In order to show the urban heat island effect was trivial, the IPCC case relies on a paper by Jones, which relies on a paper by Wang, both of which rely on the assertion that a set of weather stations in China were selected on the basis that we knew they had “few, if any” station moves or other inhomogeneities. However, we know that this cannot be true, and that the researchers involved must have known it, because it is reported elsewhere that most of the stations have no metadata, and most of those that those that do were moved often very considerable distances.
There is no doubt that a claim was made and was relied upon that cannot be true. The person making the statement could not have seen evidence of it. So far, that might be a simple error. Where it becomes outright fraud comes after it was pointed out to them, when instead of simply responding with a retraction and correction, they first ignored the requests for information (because they were from “sceptics”), obstructed attempts to examine data, and denied there was any problem. A complaint of scientific malpractice was made and investigated, but in defiance of their own procedures the inquiry was held in secret, the accuser not permitted to be involved, or to see the report. There is no public explanation or justification. No defence has been presented. Nothing.
Until now, when we see the climategate climate scientists casting around for possible defences, and coming up with nothing. Kevin Trenberth offers “So my feeble suggestion is to indeed cast aspersions on their motives and throw in some counter rhetoric. Labeling them as lazy with nothng better to do seems like a good thing to do.” Tom Wigley, Director of CRU at the time explicitly says that the accusation was correct. “Seems to me that Keenan has a valid point. The statements in the papers that he quotes seem to be incorrect statements, and that someone (WCW [Wang] at the very least) must have known at the time that they were incorrect.”
That’s a direct admission by a Director of CRU and respected climate scientist that the accusation was correct regarding a claim then under investigation as a matter of scientific fraud.
Tom can (and has) withdrawn the assertion since the emails have come out, but on what evidential basis nobody knows. The accusation appears sound, and no (public) defence has been offered. It appears to be a clear case of scientific fraud and continuing cover-up.
Based on past experience, I have no great expectation that this will persuade you. Some say “it doesn’t matter”, others that “nothing has been proved” since we have not seen the defence. But that works for me too. Believers are judged by the rationality of their arguments. But I still hope to be surprised.
An analysis of the integer overflow problem in the code:
http://www.talk-polywell.org/bb/viewtopic.php?p=31308&sid=83a1338df336afe3351b83d218528831#31308
More discussion of integer overflow and other issues here.
http://www.di2.nu/200911/23a.htm
Don’t forget to use the scroll bar to see all comments.
By the way, it’s well worth checking the ‘Harry Read Me‘ file yourself to see what the CRU programmer did on finding the bug. Did he put in new code to detect and fix the problem? Did he re-write it to avoid overflows occurring? Did he just get the program to display a warning?
No. He manually edited out the data value that had triggered the overflow, and moved on!
“Action: value replaced with -9999 and file renamed:
pre.0312031600H.dtb (to indicate I’ve fixed it)
.dts file also renamed for consistency.
anomdtb then runs fine!! Producing the usual txt files.”
Quality!
I find this particular sad because when I run across problems like this in my own field, I use the bad values as tools for improving the code. An known input set that causes invalid output is golden. I have gone to inordinate lengths to create such sets… and then to track the faults down through integration loops and god knows what else until I could identify the source of the divergence.
I just find this so sad.
Dale, likewise. It’s useful in the same way on another level, because people’s reactions to the climategate emails is diagnostic of their habits of thought. I’m always pleased if somebody manages to prove me wrong (although not entirely, sometimes) because it means I’ve learnt something, I won’t be making the same mistake in a more serious situation, and it’s data to track down what went wrong – how I came to the erroneous conclusion in the first place.
Some deny that anything is wrong, or that the emails reveal anything interesting. Some demand ever more detailed chains of evidence (which is fair enough) but demand that it all be laid out pre-digested for them, and won’t budge until it is. Some refuse to believe that this could be the real code used by climate scientists, thinking this must be scratch code as nobody would seriously use code like this for anything important – let alone publish the output so prominently in the global spotlight. And even those who do acknowledge the code’s failings refuse to consider that it may have any wider implications for climate science. So we’ve proved one line in one program by one scientist at one institution is buggy. But there’s tons more, it’s “overwhelming”, and it’s peer-reviewed too.
Climategate has not only been diagnostic of the scientific details, it is diagnostic of the political, economic, journalistic, and “scientific” process that allowed it to occur, and failed to detect it for so many years. It is diagnostic of the mindset that allows it to continue even now, in the face of all the new evidence, to set the global agenda – the political and media consensus, the emotional investment of thousands of campaigners and activists, and indeed ordinary people with no special scientific knowledge just trying to do the right thing. Those people have been betrayed.
Now it is our task to diagnose what went wrong; to trace the root source of the error. So that when the people wake up dazed, blinking in the bright light, and ask “What the hell just happened?!” somebody will have an answer for them.
There’s a lot yet to do before we can really get to that. It seems that many of the believers (like jakerman) still don’t know what’s just hit them. And most of the analysis of climategate is yet to be done.
I’m sure this will all happen again, just as it has happened so many times before. But maybe we can make it a little harder for them, next time.
PA, this may be as bad a time as any to ask you to briefly explain to a half-lay person like myself the role programming code played in this research?
Alisa,
There are lots of bits of code in the archive for many different purposes. I think this bit is from work done to try to fix the databases of weather data from all over. The work was originally done by Tim Mitchell to collect the data, put it in a common format, and then extract the useful bits for use in other projects and papers. My belief (officially unconfirmed) is that they kludged together the code roughly to generate the papers and results, and didn’t worry about quality. When the sceptics started demanding the data and programs to check it, they found they could no longer remember how to get all the programs to work to generate the results they had already published. So they gave it to Ian “Harry” Harris, to try to fix. He’s trying to work out how the original results were generated, and make it all reproducible. So it can be published without major embarassment.
There are other bits of code that are used for interpreting tree ring data as temperature, processing temperature data, and determining the parameters to be used in climate models. It’s all a bit of a mess, and I haven’t seen any useful summary sorting it all out yet.
Thanks PA, clearer now.
Another thing the layman probably isn’t aware of is just how much adjustment and processing goes into the simple-looking graphs you sometimes see.
Take temperature, for example, like in that hockeystick plot. It isn’t actually temperature being plotted, but another quantity called temperature anomaly, although even I sometimes forget to distinguish. You start with the daily min and max temperatures recorded by the weather stations. But then you have to adjust them to account for time of observation bias. This is when you read the thermometer near the time of minimum or maximum temperature in the morning or evening, and potentially get the previous day’s maximum instead of today’s. Sometimes there are missed measurements, or transcription errors, or the measurer goes on holiday for a week, or a war or revolution, or flood or hurricane. So you have gaps in the data you have to fill in somehow. Then the stations often only operate a few years before being shut down, or moved, or changed for different instruments (the mercury thermometers have in many places been replaced by electronic ones). Even simple stuff like trees growing up around the station and then getting suddenly cut down, or the paint on the housing getting dirty or mouldy and hence absorbing more sunlight, or getting repainted. And there is of course the famous urban heat island effect, where the temperature can be up to 10 C warmer in a big city than outside it. All of this has to be stitched together and “corrected”. The corrections are often as big or bigger than the change in temperature being observed.
And then when you’ve got a temperature series, you’ve got the problem that different places have naturally different temperature ranges. It varies by latitude, by altitude, by proximity to the coast, and of course rises and falls with the cycle of the seasons. All of this has to be removed – generally by calculating the average temperature at that location for the time of year (averaged over the whole of whatever data you’ve got) and subtracted to get the difference from the average for the location and time of year. This is the so-called “temperature anomaly”.
Then of course you only have the temperature at isolated weather stations – but they’re unevenly distributed, and they come and go as stations are set up or removed. (And for other reasons we don’t fully understand yet.) One of the new results coming out of the data we’ve finally got is that the number of stations and their selection has a bigger effect on the output than expected. All the wide areas between stations have to be filled in somehow. How can you tell what the global average was in 1900, when there were virtually not temperature stations in such minor places as “the Pacific Ocean”?
And so on. When people talk about “the temperature” as if it was a guy reading a thermometer, and hence pretty solid empirical stuff, it’s somewhat misleading. Most of it is a dodgy patchwork of bad data and iffy adjustments – a lot of it unchecked.
Code plays a far bigger role than most laymen think. This particular snippet isn’t actually doing that, but it’s by the same people at the same organisation, probably using the same sort of methods. You can’t rely on anything.
That’s what I was getting at, more or less. I think the disconnect in my mind is about climatologists writing computer code. Is this a problem to begin with (in other words, do these people have serious math training, beyond the rudimentary statistics they seem to teach everyone these days, except maybe for English majors:-)), or is their lack of scientific integrity the only problem?
Pa Annoyed may have his own tuppence on the matter, but I would tend to say it is more an issue of mindset than specific training. I am sure a course in numerical methods could be helpful, but a good scientist or engineer should have the wits to master this anyway.
What is needed and cannot be taught is the way in which you approach problems in code. You have to prove and prove again and keep running regression tests with an anal retentiveness level that costs you a great deal of worry, extra work and lost sleep. You have to actually *care* about your code giving the right answers such that you go through your programs as if you were proving theorems. And even then you always remain alert to anything that just doesn’t feel right and are prepared to spend hours or even days proving the matter one way or the other.
It takes many years before a body of modeling code can truly be considered trustworthy… and even then I could tighten the curl of your short and curlies with stories of things I have found in code that has been in use for up to 40 years…
It’s enough to give you nightmares.
Well this is more or less my point: anal retentiveness is the mathematicians second nature (if not the first), so these people are clearly not mathematicians. My point was not so much the actual training, but the sort of personality that would survive such training and still want to deal with this kind of stuff throughout their entire lives (I am thought of as anal compared to most people, and even I dropped out after two and a half years of engineering-level math…)
I am thought of as anal compared to most people, and even I dropped out after two and a half years of engineering-level math…
A point I keep making here and elsewhere. Engineers should be put on this. Followed by a full on engineering design review.
Engineers are the hard men of science. Absolutely brutal. Rightly so. Because lives will be lost if they are not.
Actually M. Simon, they should be mathematicians, not engineers, as the former are much worse:-) Engineers are trained to make compromises, because nothing would be accomplished if they didn’t. In the case of climatology nothing really needs to be accomplished, in fact, the less is accomplished, the less harm done.
Alisa,
Have you ever been the “victim” of a design review?
BTW mathematicians are excellent at math. What they lack is experience with the real world. Siting issues. Errors from instrument design. etc.
I have been involved in discussions at Climate Audit and it has been my experience that engineers offered the most telling criticism when it came to experimental design. What mathematician can tell you the relative merits of thermistors vs platinum RTDs? Or all the things that can go wrong with the electrical heat pumps (thermopiles) used in dew point sensors.
How many mathematicians are familiar with “gun decking” the data? Which goes to the problem of false data in the original record.
Now maybe ordinary engineers are not up to the task (I’ll grant you that). Aerospace engineers are in a different world. Lives depend on them getting it right. And it shows in their rigor.
If I had, would I still be here typing away?:-) Told you I dropped out. In any case, my point is that when it comes to climatology, there should be no design review, as there should be no design, as there should be no product. For at least the foreseeable future climatology should be pure science, as there are just too many unknowns to be messing with actual applications of what seems to be known, if anything at this point. But, as far as aerospace engineering goes, I’ll grant you that I would let Dale design the temperature-measuring instruments:-)
Ah, but it would not be I alone! Rand Simberg, Jim Bennett and I would spend a week at whiteboard and computer and yell at each other alot until we had an initial document, then we’d have a design team put together and a review team…
Alisa,
I’m a non-degreed aerospace engineer. How did I do it? Worked my way up from bench technician. Degrees are overrated (for the really motivated).
And I would still put engineers (aerospace of course) in charge of experimental design. Instruments, station siting, record keeping, etc.
You might find great amusement in looking at the Climate Audit discussion of weather instrument design. Or ocean temperature sampling with buckets. And from this motley we are supposed to be able to discern a signal buried in the noise? To an accuracy greater than the accuracy of the instruments? Which changed over time. Not to mention sampling frequency. (From what I understand daily until about 1930. Then a timed measurement plus min/max. And finally the modern era with samples every 15 minutes [possibly oftener] from electronic instruments.)
Any way – don’t give up your dream. Just try a different approach.
M. Simon: good point about degrees – I guess I wasn’t motivated enough. Besides, it was more than 20 years ago, and I am pursuing a totally different dream now anyway:-)
It is now obvious that skepticism about the AGW research, even if not about AGW itself, was very much justified.
Yeah, Dale, some people know how to have fun;-P
What you really need is an open collaboration, with engineers with experience of safety or integrity-critial applications in charge, that involves software engineers, data management specialists, mathematicians, statisticians, physicists, climatologists, instrument designers and manufacturers, station observers, historians, and so on all contributing.
The academics should carry on doing what they’re doing, in any way they like, but the data collection and processing, and the V&V process, should all be in the hands of people trained and qualified to it. They basically come along afterwards and tidy up what the academics dream up. Academics provide the ideas, engineers make it happen and make sure it’s done right.
There’s no problem with taking engineering shortcuts when it is safe to do so. But the art of the engineer is knowing when it is and when it isn’t.
And yes, the infrastructure for collating and extracting climate data from world-wide records needs a design document, and design reviews, and professional, open engineering standards to be applied. I do think that trying to make academics do all that sort of thing would stifle progress and innovation, but for that very reason academics are totally the wrong people to be doing this.
Considering the amount of money spent on climate science, and the money at stake, it’s astoninshing that it has been left to such amateurs. They might know a bit about climate, but they know virtually nothing about all the other skills needed. It would be like inviting astronomers to build commercial rocket planes for going into space.
You make sense PA, as always.