155 Responses to Public involvement in science

  1. There should be more public involvement in rocketry and brain surgery.

  2. Dikran,
    I did wonder where one would draw the line. It seems that science is one area where some argue that the public should be more directly involved, despite it being an area that would typically be regarded as requiring some expertise/training. You don’t really see similar arguments in other areas that require a similar level of training/expertise.

  3. There does seem a widespread misconception that science is easy and you don’t need to study it very hard to see that the scientists are fundamentally wrong on the most basic issues, the public debate on climate change provides and almost limitless supply of examples. On the other hand, there are members of the public who do make genuinely useful contributions, e.g. Nic Lewis, and the scientific community is generally willing to listen to what they have to say (provided they stick to norms of good scientific behaviour). This problem crops up withing science as well, when someone works in a field where they have no specific expertise. Sometimes this works well (Berkeley Earth) and sometimes it doesn’t (Prof. Essenhigh), and sometimes the change of field doesn’t have to be very large for it to go badly wrong (e.g. Prof. Salby). I think more or less anyone can get involved in science, but there are no short cuts and you have to do your homework. For some that is easier than others (for instance being comfortable with differential equations made it easier for me to write a response to Essenhigh’s paper, but I still made sure I checked everything with a genuine expert), but for some I suspect this with exceed their time/energy/enthusiasm budget.

  4. ATTP indeed, it is a bit like saying anyone can be a concert musician. This may be technically true, but if nothing else most of us don’t have the time and effort to put into practice and study to reach that level, even if we had the talent (which I certainly don’t! ;o) I’m definitely sure that not everybody could be a top-level cricketer (I have no hand-eye co-ordination whatsoever ;o). I don’t think science is vastly different from any other field of endeavour.

    We should certainly draw the line before brain surgery ;o)

  5. Willard says:

  6. Willard. That is indeed an excellent way for people to get involved, and there are a fair few “citizen science” projects of a similar nature out there (ISTR one about spotting penguins?). Kaggle, a web site for people to compete in data analysis challenges, is also open to the public and that sort of thing is very important in finding out what actually works and what doesn’t.

    I’m even (genuinely) supportive of (at least the idea behind) WUWT’s surface station project. Not too sure why the paper never got published though – perhaps waiting for the Journal of the Open Atmospheric Society to actually open? ;o)

  7. There are indeed a number of excellent citizen science projects. The Zooniverse projects seem to cover a wide range of disciplines. My impression, though, was that the suggestion was that the public could be involved in more than simply these kind of citizen science projects. I have also some criticise these on the basis that these should be properly funded and shouldn’t rely on free public labour. I think if people get some enjoyment/benefit out of this, then it’s fine. However, there may be a point at which people are being taken advantage of. Of course, no one is obligated to participate.

  8. Gary says:

    And then there’s bona fide scientific research which is rejected because its conclusions are too hopeless . See http://www.lifeworth.com/deepadaptation.pdf titled “Deep Adaptation: A Map for Navigating Climate Tragedy.”

  9. There are some very good citizen science projects out there, I’m surprised I didn’t immediately think of this one!

  10. Gary,
    And they say you can only get published in climate science if your paper is alarmist 😉

  11. Joshua says:

    Anders –

    I’m wondering if you can provide a link to an argument of the sort you are characterizing. As an abstracted characterization, yes it seems absurd to say that effectively, deep knowledge is not a precondition for conducting quality research, producing sophisticated science, etc.

    So absurd. In fact, I have to wonder if that is really what people are saying.

  12. Joshua,
    I was trying to avoid dragging an individual into this. Part of it might be what we each mean by the terms “science”, but it was fairly explicit that they didn’t mean using people as data-gathering drones. They were suggesting that we should be enabling such people to formulate hypotheses, tests, experiments – perhaps even guiding through ethics approval, peer review, publication.

  13. Joshua says:

    Anders –

    Who are “such people?”

    Is it basically an argument for explicit efforts to diversify the ranks of people who are employed to do research?

  14. ” They were suggesting that we should be enabling such people to formulate hypotheses, tests, experiments – perhaps even guiding through ethics approval, peer review, publication.”

    There isn’t really anything stopping members of the public from doing that, and there is plenty done already to enable it, for instance universities making teaching materials freely available online, academics having social media presences where you can talk to them about stuff like peer review procedures. One wonders what they had in mind?

  15. Joshua,
    I took such people to mean something like the general public. However, as I said in the post, maybe I’m misunderstanding this. It did seem, though, that the suggestion was that science engagement should be deeper than just scientists engaging with the public and should include quite a deep involvement of the public with science.

  16. JCH says:

    This reminds me of when Guv Bush proposed placing exhaust sniffers along entrance and exit ramps on freeways to test the performance of a car’s emissions system as it whizzed by. Thus allowing the big-city public to dodge an annual trip to a centralized testing center – big government.

    Everybody loved this idea, and they promoted the idiot to President of the United States of America.

    Finally, a reporter interviewed a scientist who laughed and said the notion was flat-out impossible.

  17. 1. Most of machine learning applied to uncovering patterns is a form of non-specialty science.
    2. Astronomy is loaded with citizen scientists
    3. Much of science blogging has been tainted by the delusional WUWT Talkshops that continue to win blog “awards” for science
    4. US research agencies frequently offer challenges to the public to solve problems, attracting lots of people from other countries that can expend lots of mind-power for little monetary gain
    5. Math continues to be a popular open forum discussion topic, see e.g. category theory, often spurred by puzzle-solving challenges or formal proofs
    6. Many physics forums limit the discussion of any topics outside of homework problems. The Azimuth Project forum is a rare exception to this.

  18. Jeffh says:

    I completely concur with Ken. ‘Citizen science’ is becoming increasingly popular and is making an excellent contribution to data sets that help us to understand, for example, demographic changes in populations of birds, range-expanding plants etc. But is it truly research? No. Research covers the complete gamut from possessing the relevant background knowledge to using this in experimental preparation, execution and analysis, and then to manuscript writing, submission to a peer-reviewed journal and eventual publication. Much of the ‘science’ coming from the streets is, let’s be honest here, risible. But the other problem is when people qualified in certain fields then think that gives them carte blanche expertise in other, often quite unrelated fields. Look at the number of ‘street scientists’ as well as professionals in other disciplines nevertheless lacking expertise in environmental science or ecology who keep plugging the simplistic ‘CO2 is plant food’ meme in an attempt to defend the burning of fossil fuels. Blogs and the media are full of this crap, which takes complex adaptive systems and breaks them down to the simplest common denominator. There is a guy who apprently works in IT but who has been pushing this meme on blog after blog. What he wrote was so utterly puerile that I became exasperated countering it. However, he apparently so impressed the numpties at the CO2 Coalition with his arguments that he is now ‘proudly’ affiliated with them. Bear in mind that he probably can’t tell a mole cricket from a water buffalo, but that doesn’t matter. Persistence and deep confidence in one’s convictions, even when being totally wrong, are more important. And of course saying what your target audience wants to hear.

  19. Jeff,
    Another example in the UK is Paul Homewood, who seems to be an accountant, but has become the go to person whenever someone like Christopher Booker, or James Delingpole, wants to claim that there are huge problems with the global surface temperature datasets. I’ve even had someone claim that because they have expertise with numbers that this means that they can interpret scientific data. Dunning Kruger writ large.

  20. “1. Most of machine learning applied to uncovering patterns is a form of non-specialty science.”

    err, not really. The Machine Learning is a speciality science, and you need to have done your homework to know what you are doing there as well. Not difficult to find examples of bad science where machine learning has found specious patterns because the machine learning methods were used without sufficient understanding (lots of 1990s applications of neural networks before the hype bubble burst the first time, deep learning may end up repeating that if it isn’t careful). That is why it is generally a good idea to have a collaboration between a domain expert and a machine learning expert who is interested in the domain.

  21. Willard says:

    > Much of the ‘science’ coming from the streets is, let’s be honest here, risible.

    In fairness, most of the science is crap:

    https://andthentheresphysics.wordpress.com/2018/05/03/the-pursuit-of-crappiness/

    What we’d need is a network of distrust, a system where knowledge access and production can remain as open as possible. Even contrarians could contribute. Much of their concerns are legitimate. To lack constructiveness and to burden scientists with one’s own itch is not.

    One very big step toward involving the public would be to divest from quasi-predatory publishers:

  22. The machine learning aspect is an additional eyeball, which is what a citizen (non-specialist) scientist supplies. I like to bring up the example of tidal analysis, and whether that could have been done completely via a symbolic reasoning approach. It wouldn’t have understood the physics but it would reveal and match the mathematical representation.

  23. err, no. Machine learning is essentially computational statistics. The idea that it is a tool that can be used without a really solid understanding of statistics is (IMHO) wrong headed. Very easy to get the right result (on the training set) for the wrong reason (and hence bad performance on the test set), and especially easy not to notice this if you know what the test set looks like beforehand.

  24. Marco says:

    “One very big step toward involving the public would be to divest from quasi-predatory publishers:”

    Why? Seriously, why? It’s a step for sure, but I would not call it very big, and not even big, considering how extremely few people in the public actually show an interest in reading scientific papers.

  25. libertador says:

    “Also, many people do a form of science in their daily lives. We observe things, interpret our observations, and make decisions. ”

    Well that sounds interesting. I might have constructed it the other way round as one of these operations seems to be a more basal human activity. But who am I, there may be people who started observing, interpret these observations, and decision making in their daily life after they became a scientist. I am not sure how this can be achieved, but I will not throw away this possibility to hasty.
    Seriously, it sounds pretty strange to say daily activity is a form of science and not science is a special and elaborate form of these daily tasks.

  26. liberatodor,
    I was mainly just trying to suggest that we can follow a form of the scientific method in our daily lives. I still think there is a difference between this and a more formal application of the scientific method. In a sense, we’ve probably been – as a species – behaving like this for a very long time, but only applying it to something that we’d describe as research since about the time of Galileo.

  27. BBD says:

    Palaeolithic humans had an excellent empirical understanding of ballistics but it took Newton to come up with some Laws.

  28. Willard says:

    > I would not call it very big, and not even big, considering how extremely few people in the public actually show an interest in reading scientific papers.

    I suspect you got this impressions from contrarians and not from the public in general, Marco. Even among contrarians that could change. It should if we want to integrate them into our network of distrust. At the very least contrarians would not have the excuse of hitting a pay wall.

    My own impression is that the number of independent researchers is not extremely small. If that’s not enough, consider researchers outside the Anglosphere, Western Europe, or the big capitals. The whole publishing system needs to be revised, and if publishers don’t adapt they will fall:

    Providing constructive criticisms to papers one can’t access is not quite possible. If that’s not big enough for you, I don’t know what is.

  29. My example is of applying symbolic reasoning, which is more about finding mathematical patterns. I documented a case of analytically solving the Navier-Stokes equation for a topology and also finding that same mathematical formulation emerge from a symbolic reasoning training run on real data.
    The capper was that the derived formulation is very exotic, and one that only a naive symbolic reasoner would find, simply by being able to try millions of combinations without giving up.

  30. John Hartz says:

    ATTP: You wrote:

    In a sense, we’ve probably been – as a species – behaving like this for a very long time, but only applying it to something that we’d describe as research since about the time of Galileo.

    While that may be true for Western Civilization, is it true for other civilizations such as Indian, Chinese, Aztec, Mayan, etc.?

  31. JH,
    Good question. I may well be ignoring earlier examples of people using what we now regard as the scientific method. However, it is my understanding that many regard Galileo as the father of modern science. A quick bit of Googling also suggests that Islamic scholar may have played a key role in introducing the experimental method into science in the 8th and 9th centuries.

  32. russellseitz says:

    PP
    It is cautionary that for some decades the state of the art in tidal analysis consisted in building bizarrely complex, but highly effective, cam-and-wire analog models of coastlines, basins and flows, and that in the days before computers became small enough for aircraft to carry, the Norden bombsight did its ballistic integrals by multiaxis rotation of a ball bearing via parameter wheels.

    Should AI ever reach or learn that level of geometric intuition it could kick some new climateballs on to the unlevel playing field.

    Meanwhile, though Being Digital ( check out Negroponte’s book of that name ) has increased the bandwidth of science communication, it has been more of a boon to disinformation than the signal to noise ratio of public discourse.

  33. izen says:

    ~=W
    “One very big step toward involving the public would be to divest from quasi-predatory publishers:”

    Science publishing is how business makes money from the requirement to have an outlet for research that is policed to ensure the quality of the product. (peer review?) Taking money from both publisher and subscriber extracts some of the money society invests in ‘Science’.
    It supports the elitist system that excludes the majority, and grants to a small minority access to resources and funds while according them enhanced social status and income. Similar systems operate in medicine, and Law. There used to be Guilds that performed similar roles for many past ‘professions’…

    To those excluded from these ‘clubs’ they can certainly look like elitist systems to restrict access to the status and wealth they confer to a small minority. For some with an ideological bias towards full democracy as the ONLY validating criteria for any actions with social consequences, the elitism of any profession or specialist role looks like the ‘True’ and ‘Real’ interest of the majority are subservient to the interest of the small elite group. Who may act to protect their status rather more than to enhance their contribution to society.

    AFAIK there is no serious or widespread argument that professional sport would be much improved by enabling much more public involvement. Fan participation is generally considered to best be restricted to off the field of play.

  34. Michael 2 says:

    While I agree with your view that it is easy for non-experts to assume simplicity where it doesn’t exist (in any specialised skill), your view seems somewhat incongruent with your support of child strikes for climate; youthful non-experts skipping school presumably having made an adult, aware decision which you assert most adults cannot make (for lack of expertise) yet children CAN make.

    Much better than a “child strike” would be a “child science extra day” protest, where they give up a day of vacation to learn about the relevant research. What is a child willing to SACRIFICE for “climate”? Good heavens, a day out of school is no sacrifice at all.

  35. izen says:

    @-ATTP
    ” I may well be ignoring earlier examples of people using what we now regard as the scientific method.”

    The evolution of ‘the scientific method’ was gradual, drawing a line between before and after at Galileo, or anywhere else is arbitrary.
    Much of what we now call science and the scientific method derives from religion. Most religions everywhere on the globe developed a deep knowledge of astronomy, you had to know when to sacrifice.
    Reason, logic, and arguments constructed from evidence were developed as tools in theology (Aquinas) long before they were applied in science. Discovering the material world started as a means of understanding the Creator better by knowing His creation.

  36. Michael 2 says:

    JCH says: ” to test the performance of a car’s emissions system as it whizzed by….
    Everybody loved this idea, and they promoted the idiot to President of the United States of America.”

    President Bush wasn’t elected on a platform of automated rolling emissions testing systems. He was elected after being seen drinking water, an act that resonated with his followers that also drink water.

    If you can argue from spurious correlation, soak an eye.

  37. David B. Benson says:

    Plenty of room for observers; so-called citizen astronomers, naturalists, fossil hunters, …

  38. Marco says:

    “I suspect you got this impressions from contrarians and not from the public in general, Marco. ”

    Your suspicion would be wrong. The contrarians I’ve met, in several fields, are often aware of papers, even read many, but either have a very selective reading, or simply don’t understand them/misrepresent them.

    With sci-hub, arxiv, biorxiv, several other sharing possibilities, and the possibility to ask the authors directly, a lot of papers are available to the general public despite the “paywalls”.

  39. a day of vacation to learn about the relevant research

    Could there be a clearer example that some are willing to claim that doing science hardly requires any effort or expertise.

    That being said, I welcome steps to make science more open, which should be easier nowadays with so much articles, data and software being online. I do not expect citizen scientists will help science much, I certainly could not do much next to a day job, but when people are willing to put in the work, they should be able to join the scientific community, as a matter of principle.

    When climate “sceptics” complain about pay walls just send then to Sci-Hub. http://sci-hub.tw/
    I am quite sure in over 99% of the cases they will not read the article, only wanted to attack science. People have often asked whether the data of my benchmarking study of homogenization algorithms was online, when I gave them the link they were no longer interested.

  40. John Hartz says:

    Another instance of serendipity…

    In today’s snail mail, my wife received an invitation to become a member of the Society for Science & The Public

    It’s mission:

    Society for Science & the Public (Society) is a champion for science, dedicated to expanding scientific literacy, effective STEM education and scientific research. We are a nonprofit 501(c)(3) membership organization focused on promoting the understanding and appreciation of science and the vital role it plays in human advancement: to inform, educate, and inspire.

    Since 1921, the Society (formerly known as Science Service) has conveyed the excitement of science and research directly to the public through our award-winning publications and later through our world-class science education competitions.

    Today, the Society is dedicated to providing concise, accurate, and inspirational science news and opportunities to our nearly 100,000 subscribing members, the 50,000 alumni of our competitions worldwide, and millions of unique online visitors and social media followers.

    I’m inclined to join.

  41. Willard says:

    > When climate “sceptics” complain about pay walls just send then to Sci-Hub.

    That’s, like, illegal. Also, mentioning ways to bypass paywalls concedes the point.

    Which reminds me of another reason why the public needs to be involved in science – scientists aren’t immuned to special pleading.

  42. ksreferee says:

    The thing I find disconcerting about what is being suggested comes from knowing Albert Einstein didn’t care what a person’s educational pedigree said. If they could soundly challenge his science, he welcomed it and took it seriously enough to make those challenges known, as well as to offer his falsifications to the challenges, if there were any.

    What we have in the supposed scientific field of climate science is a group of elitist bullies, who when faced by a reasonable challenge, circle the wagon proclaiming nonsense like, “The science is settled.” < That isn't science.

    Even if 1,000 scientists vehemently agree on a scientific theory, it only takes one sound differing theory backed by observation and replication to falsify any supposed known science. All science requires skepticism and to seek that which is unknown. Seeking to regurgitate the findings of another like minded person, that is a fan club, not science.

  43. izen says:

    @-W
    “scientists aren’t immuned to special pleading.”

    Can you give examples of this, and how public involvement might alter, avoid, or prevent it ?

  44. Willard says:

    > Can you give examples

    [Estr] Pay walls suck, for people should be able to read scientific papers.
    [Vlad] But I tell everyone to use sci-hub!

    [Estr] The number of independent researchers is not extremely small.
    [Vlad] But contrarians read and understand badly.

    Look. I’d rather focus on the ball here. We all want better science. We all want people to embrace a scientific worldview.

    Things are getting better. More people get educated. More people than there are jobs in science. What should we do with all these people?

    Some of them become contrarians. The prototype is a successful businessman who can afford a strange hobby, like the Auditor, NicL, RyanO, and so many more. Sure, they have their peculiarities, an entrepreneurial drive, self-reliance, pugnacity, etc. These are annoying, but useful skills. Scientists aren’t deprived of a strong personality either.

    Just-so stories won’t suffice anymore – people need to understand how science works. More people than scientists do, including those who used to study science. Those who have no scientific formation (and I’m counting engineers in that bunch) need to see scientists in action. They need to watch them talk, to read them write, and to talk with them. Scientific communication needs to adapt. It can’t only be inside baseball anymore.

    To me, science is a disciplined extension of common sense. If everyone can cook, everyone can do science. Not everyone will. That’s OK. However, education itself implies something scientific. Any method that produces knowledge does.

    I’m not suggesting it is up to scientists to mobilize amateurs. Scientists barely have time to look over other scientists’ shoulders. Why do you think most auditing stories are being done by amateurs? They have time. They have the energy. This is a resource. Let’s find ways to tap into it.

    Auditors won’t go away. If they can’t be integrated in the overall knowledge production, they’ll develop their own channels and build up their own magaphones. The GWPF is only a precursor. More will come.

  45. John Hartz says:

    Off topic, but extremely important commentaries about two sides of the ugly coin of the extreme right-wing movement…

    Why you’ll never meet a white supremacist who cares about climate change, Opinion by Rebecca Solnit, Comment is Free, Guardian, Mar 19, 2019

    Eco-fascism is undergoing a revival in the fetid culture of the extreme right, Opinion by Jason Wilson, Comment is Free, Guardian, Mar 19, 2019

  46. Steven Mosher says:

    “Just-so stories won’t suffice anymore – people need to understand how science works. More people than scientists do, including those who used to study science. Those who have no scientific formation (and I’m counting engineers in that bunch) need to see scientists in action. They need to watch them talk, to read them write, and to talk with them. Scientific communication needs to adapt. It can’t only be inside baseball anymore.

    To me, science is a disciplined extension of common sense. If everyone can cook, everyone can do science. Not everyone will. That’s OK. However, education itself implies something scientific. Any method that produces knowledge does.”

    yup.

  47. Steven Mosher says:

    “I am quite sure in over 99% of the cases they will not read the article, only wanted to attack science. People have often asked whether the data of my benchmarking study of homogenization algorithms was online, when I gave them the link they were no longer interested.”

    Yup, and that 1% can be very important.

  48. Steven Mosher says:

    “Auditors won’t go away. If they can’t be integrated in the overall knowledge production, they’ll develop their own channels and build up their own magaphones. The GWPF is only a precursor. More will come.”

    Sybils attacks.

  49. Marco wrote “Why? [divest from quasi-predatory journals] Seriously, why? It’s a step for sure, but I would not call it very big, and not even big, considering how extremely few people in the public actually show an interest in reading scientific papers.”

    For me the main reason would be that it is a misuse of (mainly) tax-payers money. However, there is also a quality control aspect. There simply are not enough competent reviewers in many subject areas to support the number of journals published in that area. While the public are generally uninterested in peer-reviewed journals, academics are not. Unfortunately not all academics, and especially students, are able to tell a good paper from a bad one, and hence time (and again tax-payers money) is wasted on research directed by nonsense that gets past peer review. There needs to be a balance between preventing the publication of nonsense and not preventing the publication of useful ideas that appear at the wrong time or the wrong context to be immediately appreciated. I personally don’t think the quasi-predatory publishers benefit that balance, quite the opposite.

  50. Willard “To me, science is a disciplined extension of common sense.”

    very much agree; would add “… in the attempt to discover truths about the nature the universe as it really is“.

  51. ksreferee,
    Are you sure you commented on the right post?

  52. Marco says:

    “Those who have no scientific formation (and I’m counting engineers in that bunch) need to see scientists in action. They need to watch them talk, to read them write, and to talk with them. ”

    Willard, the scientific outcome of research, as communicated in an article, is by far the least informative when it comes to “seeing” scientists in action. It is the end result of sometimes many years of painstakingly slow experiments, a story largely of the successes, ignoring (often by necessity) the thousands of failures on the road. The many discussions about a one single sentence or even a word in the paper are also (necessarily) hidden from the reader. And yet, these are often the most important teaching moments to learn about science. It isn’t about the “end product” (the paper), but all about the road to get there.

  53. Marco says:

    “For me the main reason would be that it is a misuse of (mainly) tax-payers money.”

    While I can certainly see there is an issue on this point (double paying), this argument is often overly simplified. For example, I work in a field where (fortunately) many industry groups publish their research. Perhaps one can come up with a rather far-fetched excuse to still claim involvement of tax-payer money somewhere and somehow, but I don’t buy that one – the industry itself may have an interest in their research being openly accessible, but there’s no misuse of tax-payer money if it isn’t. Then there are the private foundations that fund research. Many require some form of Open Access (but then don’t provide the money to do so…), but not all. Here there is perhaps a slightly more credible “tax-payer money” argument to be made, as foundations often get tax breaks, but still rather unconvincing. And the final issue I have with this argument is the fact that e.g. UK tax-payers may have funded your research, but the American, German, French, etc. likely didn’t. So, why should the “tax-payers money” be an excuse to have world-wide free access? Is this really what “the tax payer” wants and expects?

    “[snip] I personally don’t think the quasi-predatory publishers benefit that balance, quite the opposite.”

    I don’t see *any* Publisher benefitting that balance. With Open Access fees, there’s an incentive for journals to just publish whatever gets submitted, which is good for the novel ideas whose time has not yet come, but also good for the completely nutty stuff that never should have passed peer review.

  54. It seems to me that there are quite large disciplinary differences, when it comes to open access. Astronomy, and much of Physics, have been using the arXiv for more than 25 years. That means most papers (at least the accepted version) have been publicly available. Most of the data, and many of the codes are also available. There seem to be disciplines where this isn’t the case. It also seems that many don’t appreciate that these differences exist.

    There is a concern that attempts to fix the open access problem with make the arXiv non-compliant (from an open access perspective). I think this would be terrible. It would basically be penalising disciplines that have found a solution.

  55. “While I can certainly see there is an issue on this point (double paying), this argument is often overly simplified. ”

    It isn’t just that. It is questionable whether commercial publishers actually have much added value to justify the costs. Most of the work is done for free by the academics, and especially in subject areas where authors are likely to be competent with LaTeX, the cost of publishing a journal is actually very small (for example, see jmlr.org, which is a top machine learning journal, which is open access and doesn’t charge the author or reader).

    “I don’t see *any* Publisher benefitting that balance. With Open Access fees, there’s an incentive for journals to just publish whatever gets submitted, which is good for the novel ideas whose time has not yet come, but also good for the completely nutty stuff that never should have passed peer review.”

    There doesn’t need to be fees or profits made. JMLR is open access and is pretty selective in what it publishes. Peer review benefits that balance, provided there are sufficient competent reviewers. If you don’t have enough competent reviewers, there shouldn’t be additional journals.

  56. Marco says:

    Dikran, I’m sure there are cheaper solutions, and I don’t dispute that the academic publishing area has a problem with the excessive profits made by the Publishers. I do think that JMLR must have some sort of funding available to at least maintain the website. Note in that respect that Arxiv runs with a >800,000 dollar budget, which is without any organized system to do peer reviews and keep track of peer reviews.

  57. Marco, apparently USC alone spends about 30M a year on journals. 800K is *nothing*. IIRC JMLR gets a few tens of thousands of dollars a year from MIT (where it is hosted) for its running costs. The monitoring of reviews is largely automated at most journals (including JMLR) and most of the actual work is done for free by unpaid academic editors.

    The tax payers money spent on open access publishing by universities would be much better spent starting up low-cost not-for-profit journals like JMLR (IMHO), rather than maintaining the (unsatisfactory and expensive) status quo.

  58. Marco says:

    Dikran, yes, things need to be done to reduce costs, and there are plenty such options. But they are not free, someone needs to commit, and apparently, that’s not so easy to get done, or there would be hundreds of such journals like JMLR already.

  59. Sorry, UC rather than USC and $44M rather than $30M. Elsevier alone got $11M of that! source

  60. Marco they can be free for the author and reader, that is the point. Exactly what is the value added by the commercial journals that justifies the additional costs, given that the editing and reviewing is largely done for free by academics, most of the administration is automated (and has been for a long time) and there isn’t a great need for paper journals anymore?

    The reason that there are not more journals like JMLR is probably inertia and a lack of rewards for academics for this sort of work. The thing that worked for JMLR IMHO is that a large number of top researchers in the field left a commercial publisher in one go, which broke the inertia problem. Sadly you need top people to start up a top journal.

  61. Having said which, there is some evidence that starting up a new journal is very arduous and time consuming, and that operating a journal competently can be equally difficult. ;o)

  62. “To me, science is a disciplined extension of common sense.”
    Disciplined is a key word. Most people have never seen anyone seriously argue the earth is flat, but their news sources regularly carry “pro-science” intellectuals who claim it is “bigotry” to say there is a biological component to gender and they are deeply afraid of vaccinations and the GMO in corn flakes. Common sense limits the damage of undisciplined science.
    MIT released a thorough report on their investigation of what it would take for a carbon-free future. Despite the title, they find the nuclear alternative would be absurdly expensive. But all the others more so. Discipline affirms common sense.
    https://energy.mit.edu/wp-content/uploads/2018/09/The-Future-of-Nuclear-Energy-in-a-Carbon-Constrained-World.pdf

  63. Willard says:

    > the scientific outcome of research, as communicated in an article, is by far the least informative when it comes to “seeing” scientists in action. It is the end result of sometimes many years of painstakingly slow experiments, a story largely of the successes, ignoring (often by necessity) the thousands of failures on the road.

    Exactly. Hence why I suggested that those who never studied science need to see scientists in action. Those who studied science already had access to esoteric knowledge, know-how, and things better kept behind closed doors. Engineers and econometricians did not, and sometimes do not even know they don’t. If engineers and econometricians realized how science is an exploratory endeavour, many auditing miscommunications could be fixed.

    The problem I’m contemplating follows a generic ClimateBall episode. Some research is produced. A paper is out, there’s a press release. Journalists pick it up, send a quick email to the scientific team’s spokesperson to get soundbites. A newsie makes the rounds, pundits chime in, hurly burlies begin. To adjudicate them, ClimateBall players need to return to the source. Then they get stuck at some point: the paper isn’t accessible, if it is there’s no SI, if there’s a SI it’s unreadable, if it’s readable it’s in proprietary code, if it’s open source then it needs to be ran. Only if I succeed to do all this then I could go back to the text and fully understand the editorial choices the authors made.

    It’s more complicated than that – this varies from one field to the next, one can’t simply run mainframes in one’s cell phone, technical texts are hard to read, etc. My point is simply that the process is not simple. Most don’t do this. Most ClimateBall episodes peter out before that – our recent peddler does not seem to realize that the MWP wasn’t in 1400, which makes the narrative sold by our Beloved Bishop a bit moot.

    In a nutshell, the problem is quite simple: we’re bombarded daily by research that is mostly undecipherable. Yet the interest is there, more than ever. If it wasn’t for some study I haven’t read somewhere, I would not have the same party conversations. Folks around me heard about coral reefs, Antarctica, the Polar Vortex, carbon taxes, rewilding, whatnot. How can they improve their knowledge? They’re not all patzers or schmucks. Some of them were researchers themselves, most of them are not contrarian.

    There’s an obvious gap, a gap that should be less difficult to fill than then one between objective reality and the subjective experience.

    ***

    Imagine some jocular critic JC who gets a knack for some scientific topic. A bit like most of us. JC develops an expertise and creates enough research to get published. It goes from tweets, to forum comments, to blog posts. JC gets feedback, mostly critical. JC persists, but get stuck. JC is not one of the inner circle, and it shows. What are the odds that JC turns around and publishes in a contrarian outlet?

    Nobody’s born contrarian.

  64. Willard says:

    > Common sense limits the damage of undisciplined science.

    It also limits damage done to a thread by disciplined baiting or peddling, JeffN.

    Nuclear drive-by done. One drive-by per thread. Like everyone else.

  65. The Very Reverend Jebediah Hypotenuse says:


    A paper is out, there’s a press release. Journalists pick it up, send a quick email to the scientific team’s spokesperson to get soundbites.

    Guess the fraction of journalists with any scientific training…

    Scientists who care need to help journalists as much as they dare.
    https://www.nature.com/articles/d41586-019-00883-7

    Otherwise, we are at risk of never-ending chocolate hoaxes:
    https://en.wikipedia.org/wiki/John_Bohannon#Intentionally_misleading_chocolate_study

    And “knowledge-based journalism” sounds like a good idea to me.
    https://www.frontiersin.org/articles/10.3389/fenvs.2017.00094/full

    “Informing the News: The Need for Knowledge-based Journalism”
    (Book) By Thomas E. Patterson


    Nobody’s born contrarian.

    Being a really good contrarian is even more of an investment than being a really good scientist.
    Maybe that’s why there are so few really good contrarians.

    Nevertheless – We desperately need really good contrarians:

    If we all worked on the assumption that what is accepted as true
    were really true, there would be little hope of advance.

    – Orville Wright

  66. John Hartz says:

    The Very Reverend Jebediah Hypotenuse:

    I would add that there is a world of difference between a traditionally-defined contrarian and a peddler of pseudo-science poppycock such as a typical climate science denier..

  67. Joshua says:

    Jeff *

    … “pro-science” intellectuals who claim it is “bigotry” to say there is a biological component to gender and they are deeply afraid of vaccinations and the GMO in corn flakes.

    Are you saying that those groups are largely congruent?

    If so, please show some evidence.

  68. Steven Mosher says:

    “Exactly. Hence why I suggested that those who never studied science need to see scientists in action. Those who studied science already had access to esoteric knowledge, know-how, and things better kept behind closed doors. Engineers and econometricians did not, and sometimes do not even know they don’t. If engineers and econometricians realized how science is an exploratory endeavour, many auditing miscommunications could be fixed.”

    Well, I will say this.
    From the outside ( auditing) science appeared to be one thing.
    From the inside ( working with Muller, Curry, Robert Way, Rohde, Menne) it is different.
    way different.

    Jon Claerbout ( kinda my hero ) once said something like ” a science paper is the advertisement for the actual science performed”

    There are reams and reams of explorations I’ve done that I will never publish.
    Lots of dead ends. Lots of obvious things proved, Lots of projects 95% complete with no real way to finish….and at the end of this 10 years “playing” at doing science..a flesh and bones lived respect for what actual working scientists actually do. I couldnt do it full time or for a lifetime. No way. Doing it part time or as a hobby keeps me off the streets ( like night basketball) and so slightly less havoc is created.

    I used to get pissed when gavin would say do your own science. Then I tried. Here is a clue:
    Throwing stones from the outside is a fuck of a lot easier than gathering the rocks that will serve as a good foundation to build something tangible and useful to others. I think Gavin’s advice ( or challenge) was brilliant.

    Related: a little more on archives, power and FOIA

    “There is no political power without control of the archive, if not of memory.
    Effective democratization can always be measured by this criterion:
    the participation in and the access to the archive, its constitution and its interpretation.
    (Jacques Derrida: Archive Fever, 1994)

    https://depotdrengen.wordpress.com/memory-monuments-and-archives-some-reflections-on-secrecy-and-access/

  69. Willard says:

    > there is no political power without control of the archive, if not of memory.

    Counterpoint: PatrickM is still arguing that he’s a founder of Greenpeace. He even shows that by appealing to documents that may not cohere with his claim:

    Since he can’t provide incorporation papers and isn’t part of Greenpeace anymore, he has no leg to stand on. Except that he went on Tucker’s and got retweeted by teh Donald.

    So brace yourselves – we’ll soon have Q anons into our ClimateBall fights.

  70. Steven Mosher says:

    Yes interesting Willard, I had gone looking for the incorporation docs.

    I’m not sure merely making the argument that he was founder gives him any kind of real power. But then I would be quibbling about the meaning of the word power, by saying “real”. meh.

    lets consider this and see if we can agree on some cases:

    Absence from the archive, being erased from the archive, being excluded from making history, being exlcuded from writing history, not being able to celebrate your history, or being relegated to the margins is not the epitome of power.

  71. Michael 2 says:

    Joshua suggested “This is timely” an article about social media and child vaccination.

    Thank you; it was indeed interesting and a welcome change of topic. I admire Kids Plus for diving into this phenomenon. One need not look far to find coordination of like-minded persons against opposing like-minded persons.

  72. Joshua says:

    Michael 2

    One need not look far to find coordination of like-minded persons against opposing like-minded persons.

    That seems a bit cryptic to me.

  73. John Hartz says:

    Joshua: Tales from the Crypt perhaps.

  74. Dave_Geologist says:

    Gary,
    And they say you can only get published in climate science if your paper is alarmist

    Although, ATTP and Gary, the “Sustainability Accounting, Management and Policy Journal” doesn’t really sound like a science journal. So did he submit a scientific paper to a non-scientific journal, or is the topic of the paper not science?

    I’m slightly surprised by the other reason, that there are no “publications on the implications of ecologically-induced social collapse, globally, upon which to build”. Surely there are. Or at least, there are papers on the impact on the Syrian civil war, likely future impacts on agriculture for various key crops, livability within wet-bulb-temperature constraints, impacts on ecosystem services and extinctions, some of which are global. The Punjab or Dubai becoming uninhabitable for those without access to reliable 24/7 aircon would surely qualify as social collapse, or at least you wouldn’t have to join many dots to come up with that consequence. And if you were only replicating past work that stopped at wet-bulb temperatures and ignored social consequences, SAMPJ would definitely be the wrong journal.

  75. Dave_Geologist says:

    Not entirely true ksreferee (BTW why do certain groups of people always choose names like Auditor, Referee, Honest Broker – it doesn’t work with me: in fact I’m triggered because I remember precisely how democratic the German Democratic Republic was).

    Even if 1,000 scientists vehemently agree on a scientific theory, it only takes one sound differing theory backed by observation and replication to falsify any supposed known science.

    If that were true, the sight a single bird or aircraft in flight would disprove the law of gravity.

    On the substantive point, contrarians have had three decades to come up with a sound differing theory, and have managed only a big fat zero. Remember, the differing theory also has to show consilience like the existing theory. IOW not just explain the supposedly aberrant point, but also the thousands of existing observations which fit the existing theory. A couple of dozen mutually contradictory hypotheses that (sometimes; usually not) individually explain one thing, but not a gazillion other things or are contradicted by those things, ain’t worth a hill of beans. CO2 that behaves differently in the wild, anyone? At some point, sensible people realise that There’s No There There. That the Earth really, really isn’t Flat.

    Of course there will always be Flat-Earthers.

  76. Dave_Geologist says:

    Willard, I find it very unusual these days for the SI not to be freely available, even if the paper is paywalled. There is often a delay of days or weeks before it’s uploaded. By proprietary software you presumably mean the likes of Matlab, as opposed to R. A really serious citizen scientist could buy their own copy of Matlab for £1800 (plus an annual maintenance fee if you want support), or £720 p.a. including maintenance. The licence covers a single user on up to 4 PCs. GNU Octave is free and largely syntax-compatible. If the citizen scientist is expecting a nice GUI app bundled with the data in an Excel spreadsheet and can’t cope with scripts, that citizen scientist should perhaps consider whether they are up to re-analysing the data anyway (Dunning-Kruger applies). If the citizen scientist is in receipt of thousands or hundreds of thousands of dollars in Heritage funding, he has no excuse for not buying Matlab. If the University has a Matlab licence, and the researchers have decades of Matlab routines and libraries to call on, it’s perfectly reasonable for them to stick with Matlab rather than risk an undetected or hard-to-debug error slipping in through a conversion to Octave or R. Come to think of it, a useful citizen scientist project might be to do just that and exhaustively test it on real datasets. Then, because they’re in it for the science, they could hand it back to the researchers. Of course, that presumes that there are no Bad Actors. That distracting climate scientists by making them convert between software and write Help files that a child could follow is a genuine effort to make science accessible, as opposed to a ploy to make the scientists less productive. The extent to which the SI is downloaded and used, let alone novel results published, would suggests that there are indeed Bad Actors out there (witness dikran’s experience).

  77. Dave said:

    ” If the citizen scientist is in receipt of thousands or hundreds of thousands of dollars in Heritage funding, he has no excuse for not buying Matlab.”

    Hypothetical at best. Better: Analytical closed-form solution to an accepted physics model. Fixed parameters that anyone can input into a spreadsheet. That’s what anyone can do, and even then trying to get others interested is difficult.

  78. Scientists should be free to use whatever programming environments suits them, I don’t see why they should avoid using something like MATLAB just so that potential citizen scientists have a lower hurdle to using the SI material they provide. Independent reproduction (rather than merely re-running the same scripts) is better for science in the long run anyway. Making reproducible code is not nearly as easy as some suggest.

  79. Independent reproduction has a lot in common with the application of pseudo-code that is common in computer science texts. The underlying programming language is irrelevant to the algorithm or physics formulation that is being explained.

  80. Dave_Geologist says:

    Analytical closed-form solution to an accepted physics model. Fixed parameters that anyone can input into a spreadsheet. That’s what anyone can do, and even then trying to get others interested is difficult.

    Paul, as we’ve discussed before, there are real-world situations which can’t be reduced to a closed-form solution of an accepted physics model. In most of what we’re talking about, the problem is not the use or non-use of accepted physics, but that properly representing the real world using accepted physics results in situations where there is no closed-form solution. Sometimes Nature just doesn’t cooperate. I’ve seen such situations in my professional life, and others have been discussed on this blog, where strict adherence to models with closed-form solutions restricts you to a world that is so different from the real world it might as well be an undiscovered exoplanet. For example, economic models that don’t properly represent the carbon cycle and reach optimistic conclusions about mitigation requirements because their model draws down atmospheric CO2 an order of magnitude faster than the real world can.

    In those situations you have three choices:

    1) Model the fantasy world using closed-form solutions and congratulate yourself you’ve avoided any numerical pitfalls. But not be surprised when people say “meh, it’s just a fantasy world, look how drastically it diverges from reality if we omit the most recent ten years of data and hindcast”.

    2) Model the real world numerically, while being cognisant of the potential pitfalls and taking steps to mitigate them. Independent reproduction is better for that than Auditing, for the pitfalls as well as the science. Auditing will just find the pitfalls that reside in library solvers or hardware differences. Different code, different solver, same or similar data may identify structural problems.

    3) Give up.

  81. Willard says:

    > I find it very unusual these days for the SI not to be freely available, even if the paper is paywalled. There is often a delay of days or weeks before it’s uploaded. By proprietary software you presumably mean the likes of Matlab, as opposed to R. A really serious citizen scientist could buy their own copy of Matlab for £1800 (plus an annual maintenance fee if you want support), or £720 p.a. including maintenance.

    Just saw your comment, Dave.

    By serendipity, here’s what I just saw:

    The International Journal of Public Opinion Research. No publicly available SI. That’s unacceptable. This is a common experience. I get it daily. Things are better than they were. Some fields improved a lot. Climate scientists got the memo. This norm should not be dictated by the publisher or the field. Publicly funded research is guided by clear principles. It might be time to enforce them.

    As for the price of software licenses, you’re making my case. I will simply observe that you associate being a serious researcher with being able to afford something that costs twice the price of a gaming laptop. And that’s just for one program – there are others to buy, as many as researchers use.

  82. John Hartz says:

    The Earth Day Network answers the question, What is citizen science? here:

    https://www.earthday.org/earthchallenge2020about/

  83. Dave said:

    “In most of what we’re talking about, ..”

    I’m not talking most, but referring to the consensus that having closed-form solutions are an important part of math. That an analytical solution to Navier-Stokes is offered as a $1M Millenium problem is partly a nod to a math challenge but also to the fact that having such a closed-form solution is a huge benefit to solving real world problems. I’ve been publishing results of my own solution to an N-S formulation and can tell you that it cuts down on computation by many orders of magnitude.

  84. As far as the laws of mathematics refer to reality, they are not certain; and as far as they are certain, they do not refer to reality

    [Albert Einstein]

    seems relevant somehow.

  85. Willard says:

    > I’ve been publishing results of my own solution to an N-S formulation […]

    No kidding.

    Drive-by done, Web. Go find another thread somewhere else.

  86. I placed this in the wrong thread, my mistake:

  87. “Earth Challenge 2020 Marks UN World Water Day by Announcing the Six Citizen Science Research Themes for its 50th Anniversary Year, Press Release, Earth Day Network, Mar 22, 2019”

    I’m not a citizen scientist but deal with these public research initiatives regularly. Lots of prep work, typically not much pay-out per the effort expended, and often lots of strings attached. Interesting is that they usually get the most interest from the near-third-world countries, who are desperate to get any kind of funding and can put lots of effort in to the work. These are typically NASA or DOE initiatives, or some company like IBM, who I had a conference call recently on their community grid. The payoff on the latter is that you can get free computational services.

    This one appears not to have any funding or awards included for addressing the research questions but do have a content management server set up. I wish them good luck getting something going.

  88. Steven Mosher says:

    “This is a common experience. I get it daily. Things are better than they were. Some fields improved a lot. Climate scientists got the memo. ”

    Yup. huge change.

  89. Dave_Geologist says:

    Willard, obviously there are different forms of citizen science. SETI@home, for example, makes itself deliberately joe-soap-friendly. People who use Matlab don’t do it to exclude citizen scientists. They use it because it allows them to work faster and more efficiently. If you insist all publicly funded science uses simple, freely available software that requires no learning curve, you’ll have to employ a lot more scientists to get the same results. I’m all for open source, and indeed am typing this on Chromium, on a home-built computer running Kubuntu Linux 18.04 (I used to prefer the bleeding-edge but as I got older, I decided life was too short and now stick to the LTS versions). Someone who really, really wants to replicate a model built and run in Matlab can download, install and learn Octave. It’s free, and on Ubuntu there are precompiled binaries, libraries and front and back ends which can be downloaded and installed at the click of a button. You don’t even have to touch a command line (although if you’re scared of the command line, you won’t get far with it). There’s even one called octave-missing-functions that lets you check whether there is some needed Matlab function that’s not available in Octave and reports. The limitation for a lot of this stuff is not the licence fee. It’s that even with the software, it’s just too hard unless you have the relevant expertise and training. Just as fixing a modern car, doing a root canal or diagnosing a severe headache is hard. If you’re not competent to debug a conversion to Octave, you’re probably not competent to be doing that particular piece of citizen science in the first place.

    And if you’re a serious citizen scientist, intending to publish in the peer-reviewed literature, you’re probably going to spend so much time on it that it becomes your main hobby. Lots of people spend £720 p.a. on their hobby, or £1800 on, for example, a gaming computer or skiing kit. I’d probably go for the one-off purchase, because I suspect customer support isn’t much help unless you’re running something common like Red Hat on an HP Z-series workstation, so you’d only be using it for version updates. Except I wouldn’t. I’d use Octave because I like finding new things and would be quite pleased if I had something to debug.

  90. “Someone who really, really wants to replicate a model built and run in Matlab can download, install and learn Octave. “

    In general, afraid not. Matlab Simulink is used in many control system designs, with companies paying $$$ to MathWorks for licensing fees and support contracts. I spent ~20 years surrounded by controls engineers and R&D staff using Matlab, applying it myself when appropriate.

  91. Willard says:

    > Lots of people spend £720 p.a. on their hobby, or £1800 on, for example, a gaming computer or skiing kit.

    I count 29 countries in the world with a bigger average monthly income than that.

    Would you fork 3200$ and up to run Nordhaus’ 22 lines of code:

  92. PaulPikite wrote “Matlab Simulink is used in many control system designs,”

    Simulink is more for engineering, rather than science, so I don’t think that is relevant.

    Octave is free and I completely agree with Dave_Geologist that “If you’re not competent to debug a conversion to Octave, you’re probably not competent to be doing that particular piece of citizen science in the first place.”. There are no short cuts in science, just like there are no short cuts in philosophy, or social science, or becoming a concert pianist or being a professional cricketer or any other field of excellence (and that is what we are talking about when we are trying to reproduce scientific results, even the ones that turn out to be incorrect).

  93. I once got a paper in a top journal from one line of code. Lines of code is not a good metric of quality.

  94. Well, essentially one line of code that could have been added to someone else’s program, but I replicated their method myself as it is a good way of helping you to make sure you actually do understand how it works.

  95. … oh, and you need MATLAB ;o)

  96. Steven Mosher says:

  97. “Simulink is more for engineering, rather than science, so I don’t think that is relevant.”

    Many scientists and engineers would rather think in terms of a graphical dataflow than a structured algorithm. In a lab environment especially, experimental physicists often use dataflow tools such as Simulink and LabView to do signal processing on data sets or for doing various lab automation tasks, i.e. data acquisition, etc. It also has code-gen capabilities such that it can be linked into other tools.

    This is a recent paper describing how you get what you pay for:
    A Comparative Study of Numerical Analysis Packages: http://www.ijcte.org/vol10/1201-SE0010.pdf

    And issues with Octave from last year (not too bad, imo)
    GNU Octave: some mistakes and difficulties: https://www.longdom.org/articles/gnu-octave-some-mistakes-and-difficulties.pdf

    The point to consider is that if you do find an issue with a tool, you can likely get it resolved more quickly with a support contract. And if you really want to see the user activity and what the application area is, check out the forum
    https://www.mathworks.com/matlabcentral/answers/?s_tid=gn_mlc_ans_ans

  98. Dave_Geologist says:

    Willard, as I said, you don’t need MATLAB (for lots of stuff). Yes there are lots of poor countries in the world. But very few of them have the sort of citizen scientists we’re talking about among their average earners. For starters, they couldn’t afford the hardware. But citizen science in its usual meaning, like Auditing, is largely a rich-world game. I wouldn’t pay a penny for GAMS. Not because I think it isn’t good – I presume it is. Because the problem with most such models AFAICS is not that you need clever optimisation solvers and management-friendly output, but that it’s rubbish-in, rubbish-out. Too much has to be invented or assumed, too little real-world physics is incorporated, and the tantalising attraction of simplifying to a closed-form solution that fails to match reality is the final straw in at least one paper we’ve discussed on here.

    Per dikran, 22 lines of code could take days to run on a supercomputer if it calls library routines compiled from thousands of lines of code and pre-formatted datasets containing millions of observations.

    Paul, AFAICS Simulink is for engineering design. You could always try Scilab if you need that functionality. Which you probably don’t to run a climate model from a published paper. Not that I see much point in that – unless you use a different dataset, you’re just Auditing.

  99. Willard says:

    > Lines of code is not a good metric of quality.

    If I was shooting for quality, I’d go with multiple metrics and common standards. The opinions of experts might be necessary, as software sizing is an art form. Estimating costs of software production is a bane, (Mental note: it’s a good example to show that uncertainty is no one’s friend.)

    One could argue that Bill’s code is efficient, that it simplifies previous models, or else. In both cases I’d expect a formal demonstration, ideally using a model checker. We’re not there yet.

    All I’m asking for now is being able to run Bill’s code without having to sacrifice my next summer vacations.

  100. Code quality has a large subjective component, there is a limit to what you can get out of any metric. At the end of the day, an important factor is whether the code is intelligible by other human beings, which a computer can’t tell you (yet). Getting it published in a top journal and cited a couple of hundred times was more than good enough for me! ;o)

    Code quality is always a matter of diminishing returns, and for research software, moving on to the next research idea is a high priority, because the rewards for academia do not currently include software quality and society may want that, but they are apparently unwilling to pay the costs. If you want to be able to run Bills code without spending any money, then we need Bills employers and funding agencies to pay the costs of the extra work that would entail for Bill.

  101. Also worth mentioning Steve Easterbrook’s excellent video/paper about improving the software engineering of climate models. IIRC the basic result was that was unlikely to happen as the people with the software engineering expertise didn’t have the maths to understand how the code actually worked, and hence would just get in the way. ISTR it has been discussed here before?

  102. “The point to consider is that if you do find an issue with a tool, you can likely get it resolved more quickly with a support contract.”

    LOL, YMMV on that one (not alluding to any specific provider of mathematical programming environments ;o)

  103. Willard says:

    > Code quality […] Code quality

    That’s a lot of things on something that is irrelevant to my point, Dikran.

  104. Willard says:

    > If you insist all publicly funded science uses simple, freely available software that requires no learning curve, you’ll have to employ a lot more scientists to get the same results.

    Alternatively, you hire scientists who can use simple, freely available software.

    As an added bonus, you compel them to abide by the reasons why they’re hired in the first place:

    The creation of new knowledge lies at the heart of the research university and results from tremendous investments of resources by universities, federal and state governments, industry, foundations, and others.

    The products of that enterprise are created to benefit society. In the process, those products also advance further research and scholarship, along with the teaching and service missions of the university. Reflecting its investments, the academy has a responsibility to ensure the broadest possible access to the fruits of its work both in the short and long term by publics both local and global.

    Faculty research and scholarship represent invaluable intellectual capital, but the value of that capital lies in its effective dissemination to present and future audiences. Dissemination strategies that restrict access are fundamentally at odds with the dissemination imperative inherent in the university mission.

    https://www.arl.org/storage/documents/publications/disseminating-research-feb09.pdf

    An optimistic prediction – soon enough academics will rediscover versioning systems.

    GIT ALL THE SCIENCES!

  105. Willard: (i) part of that was me agreeing with you (“The opinions of experts might be necessary”). (ii) aren’t I allowed to have a point of my own in a discussion?

  106. Dave said:

    “Paul, AFAICS Simulink is for engineering design. You could always try Scilab if you need that functionality. Which you probably don’t to run a climate model from a published paper. Not that I see much point in that – unless you use a different dataset, you’re just Auditing.”

    Physics is half theory and half experiment. Like I said, there are plenty of lab environments that use Matlab/Simulink (or LabView, etc) to do data acquisition or lab automation. Where I worked, I would refer to it as LabMat. This discussion was started because you claimed that Octave could run any arbitrary Matlab project — that point will never happen of course.

    Beyond that, Simulink is seamlessly integrated within the Matlab environment, and its common to find a pure Matlab algorithm that gets wrapped in a Simulink dataflow component for whatever reason. They may want to demonstrate the dataflow, or to do some signal processing, include pop up graphics windows, etc.

    This is funny, a comical ENSO analysis in Matlab
    https://www.mathworks.com/help/curvefit/custom-nonlinear-enso-data-analysis.html

  107. Steven Mosher says:

    willard I will note that over the past ten years the quality of the dinosaur’s arguments against free and open science has diminished considerably.

    It’s just a matter of time. Kids come up in school learning python and R and
    GIT. In the same way climate denial will vanish with the death of rich old white skeptics, open science denial will die as old fart geologists immigrate to the what for the rest of us is the wrong side of the grass.

  108. Steven Mosher says:

    funny story about Knuth using a female name to get better per review

  109. izen says:

    @-SM
    “I will note that over the past ten years the quality of the dinosaur’s arguments against free and open science has diminished considerably.”

    Never mind the quality, follow the money.
    I will take ‘Open Science’ seriously when research methods/results cannot be hidden/biased because it is ‘proprietary’ or ‘commercially sensitive’.

  110. Willard says:

    > aren’t I allowed to have a point of my own in a discussion?

    You’re right. Sorry.

    I should try to channel my (fairly new) inner Jonathan.

  111. Dave_Geologist says:

    Paul, I never claimed that Octave could run any arbitrary Matlab project. Read my comments again. However, it does try to keep up to date with Matlab, so given the timeframe and collaborative nature of most climate models, you’d probably want to match Matlab functionality of five or ten years ago, not something added this year. And as I said, if you want to know whether there’s missing functionality before embarking on the effort, There’s An App For That.

  112. Dave_Geologist says:

    Willard the scientists can use simple, freely available software. But in some cases, for some tasks, it just doesn’t cut it. Or would make things so laborious and inefficient you’d need many times more scientists and vastly increased (mostly taxpayer) funding. Them’s the breaks. None of which means that the scientists concerned are in breach of the obligations set out in your chosen version of the Academy.

  113. izen says:

    When the subject of ‘Public involvement in Science’ is raised it is very rarely about making advanced computational software (or observational hardware), available to the public.
    It is usually ‘code’ for scientific research to be directed towards issues, problems, and subjects that the public cares about or is of direct concern/impact upon them.
    Instead of allowing scientists to fritter their time and expertise on subjects of no public relevance, like exoplanets or pre-Cambrian fossils. Research that might capture the interest of a small niche, but has no benefit or social relevance.

    Public involvement in Science is invariably part of the argument that research SHOULD be based on the concept that the worth of science is how much benefit (financial or material) the science can provide to the individual or economy.
    It is not about the ‘Public’ being able to access or deploy advanced programming tools.

  114. I don’t have any allegiance to Matlab / Simulink as I prefer programming in source, but only pointing out that a significant fraction of the scientific population may prefer dataflow tools. And engineers are likely weighted more toward this as well, simply due to the teaching via schematics early in their education.

    In terms of software engineering for scientific models, a nice direction is the use of typed, declarative languages such as Haskell. . The benefit here is the way that library components can be reused, so that the possibility for contributions from everyone is opened up.
    It’s very easy to get left behind here as it takes a different mindset than what the typical programmer is used to.

  115. Steven Mosher says:

    “I will take ‘Open Science’ seriously when research methods/results cannot be hidden/biased because it is ‘proprietary’ or ‘commercially sensitive’.”

    err open science is the push to eliminate closed science.

    1. free and open code
    2. free and open data.

    some of us purists believe we are not rationally obligated to even consider any science that is not free and open.

  116. izen says:

    @-SM
    “some of us purists believe we are not rationally obligated to even consider any science that is not free and open.”

    Your purity excludes a large proportion (the majority?) of clinical research.

  117. Willard, no problem. I am substantially in agreement. I’ve been meaning to learn Python for a while, mostly for distributing my tools more widely. A sign that you are doing useful research is that someone else picks it up and runs with it – best not to make that any harder to do than need be (I would also venture that it is a good reason to avoid “proof by intimidation” etc and make your papers as easy to understand as possible – but no easier ;o).

  118. Yea, “1. free and open code
    2. free and open data.”
    seems to kind of overlook the idea that many of the big breakthroughs in physics are essentially described as a pseudo-code (i.e. a non-executable math theory) of an idea. Why someone would not adequately describe their theory is beyond me. Are people supposed to guess as to what the theory means?

  119. The Very Reverend Jebediah Hypotenuse says:


    1. free and open code
    2. free and open data.
    some of us purists believe we are not rationally obligated to even consider any science that is not free and open.

    Sigh.
    By that standard, science did not exist prior to the development of op-amps, digital memory media, and executables.

    I love me some tribal identity-politics spiced up with arrogant dismissal of almost all off science as much as the next guy, but with that comment, Mr Mosher declares that he hasn’t the faintest idea that science is not equal to open-source software engineering.

    Pauli didn’t need a computer to predict the neutrino.
    Cowan and Reines didn’t need a computer to test that prediction.

    Weak neutral currents were conjectured by Salam, Glashow and Weinberg, with no code – and confirmed the same year, in a in the Gargamelle bubble chamber at CERN. Before there was an internet.

    Higgs didn’t need a calculating engine to anticipate the zero-spin, even-parity particle / field that bears his name.
    And while computers were indispensable for examining the data from Atlas and the CMS at the LHC – No one (not even purists!) is rushing in to examine the code developed at CERN, and there is (as yet) no book on Amazon called “Higgs-gate, the Boson Letters”.

    Theoreticians do not publish their findings as APIs.
    Experimentalists know that code is useful for reducing and analyzing data, but also that obtaining good experimental data – and knowing what even counts as good experimental data in the first place – is usually a much more difficult challenge than coding the analytical tools. Coding is what grad students are for.

    Self-identified purists who believe that are not rationally obligated to even consider stuff are a dime per dozen.

    Reading Mr Mosher on now to be a science purist is like reading Dr Curry on how to avoid science advocacy.

  120. That looks like a good scientific discussion on the new Bayesian estimate article, but I’ll leave it to those who know what they’re talking about. I just read this, and have also read Goodell’s book – https://www.climaterealityproject.org/blog/talking-climate-and-health-water-will-come-author-jeff-goodell – which is excellent for an overview with observed specifics. This quote from his Rolling Stone article about going along on a research voyage to Antarctica makes observations including who and what are scientists and science, and what can we see and learn.

    For me, this was a voyage of discovery in a more personal way, an encounter with a world that is as distant from my everyday experience as a trip to Mars would be.

    Among my discoveries: I learned that elephant seals have very bad breath, and are both terrifying and hilarious when they arch their backs and roar at you. I walked among a waddle of Adelie penguins and discovered they have no fear of humans, and in fact seem to think they are superior to us in many ways. I watched albatrosses follow our ship and understood why sailors once believed (and perhaps some still do) that they carry the souls of dead sailors with them. I learned that Skuas — a gull-like seabird — have bad attitudes and will steal food right out of your hands, and that albino Orcas really do exist. I discovered that if you spend enough time staring at the ocean, you will see something interesting — including, if the conditions are just right, a green flash as the sun falls below the horizon. I learned that nine-tenths of an iceberg is beneath the surface of the water. I discovered that pancake ice really does look like a million pancakes scattered over the surface of the sea, and that ice shelves, like old men, show their age by growing craggy and crevassed. I learned that katabatic winds off the glaciers are terrifyingly cold. I discovered that the Antarctic Circumpolar Current is the largest current on earth, with five times the volume of the Gulf Stream. I found that the language of geology has its own poetry (“dilated tills” “basal drag”), and that the blue of the sky and the blue of the sea are as different as the sky and sea themselves.

    I also learned plenty about climate scientists and the work they do. Like the rest of us, they capable of making mistakes, pushing flawed hypothesis and over-interpreting data. Money matters a lot to them, but not in the ways that climate deniers think (it’s all about research funding, not ski condos in Aspen). I learned that some scientists can read sediment cores like a book, with each chapter full of new characters engaged in a mighty struggle to survive on our ever-ever-changing planet. I learned that science is not only hard and often dangerous work, but that it is also impromptu, improvisational and weather-dependent. And that on a ship like the Palmer, scientists are only as good as the marine technicians and crew members who are working with them. Most importantly, I learned that the best scientists are radical and fearless in ways that few outsiders can understand or appreciate. They are heroes of our time.

    There’s a lot more, but that’s already too long. Good coherent reporting in my opinion (25 March, 2019) https://www.rollingstone.com/politics/politics-news/journey-to-antarctica-thwaites-goodell-return-812863/

  121. dhogaza says:

    Mosher:

    “some of us purists believe we are not rationally obligated to even consider any science that is not free and open.”

    As a thought experiment imagine Mosher dropped into the power hierarchy of Imperial Japan a day after Nagasaki. “But emperor, the Americans haven’t published their results where they are free and open for me to read them, we are not rationally obligated to even consider the science that led to the destruction of these two cities! And have you read the e-mails where some of the researchers secretly called Teller an asshole? Quick, I need to publish a book!”

  122. John Hartz says:

    ATTP: Fodder for a new OP. about the scientists’ role in communications..

    https://link.springer.com/article/10.1007%2Fs10584-019-02414-9

  123. For verifying research findings, taking someone’s source code and evaluating and/or reworking it can lead to common mode errors. But taking the germ of the idea and evaluating it from scratch or “clean-room” provides independent substantiation.

  124. Willard says:

    > Your purity excludes a large proportion (the majority?) of clinical research.

    See also:

  125. Steven Mosher says:

    “For verifying research findings, taking someone’s source code and evaluating and/or reworking it can lead to common mode errors. But taking the germ of the idea and evaluating it from scratch or “clean-room” provides independent substantiation.”

    1. me providing the code and data does not FORCE YOU TO USE IT or even look at it.
    2. if you believe the risk of common mode error is high. dont look at it.
    3. Having the code available, doesnt Force you to avoid doing things from scatch.

    You see, you guys miss the whole point. You think that having the methods ( the actual code IS the method) forces you to do something. it doesnt.

    Its truly bizarre.

    It would be like saying I should not supply data because it would keep you from going out and collecting your own data.

  126. Steven Mosher says:

    “Sigh.
    By that standard, science did not exist prior to the development of op-amps, digital memory media, and executables.”

    err no. Think harder, bube. Code is my short hand for method.

    if you dont show your method or data, then I am not rationally obligated to consider your work.
    It very well may be science, but I am not rationally obligated to consider. In short, if
    you dont show your work, I’m not gunna waste my time.

    pretty fucking obvious that we got where we are today because people actually shared methods and data. Now, more and more our methods come down to running some code. hence the
    simplified version: share the code..

    Grow up and share the methods.

    Science wont crumble if you share the methods.

    90% of the time if you share the code, folks wont use it anyway.

    as for data. yes yes there are exceptions where some fields have to be anonymized.
    Like I said before dinosaurs are gunna search for exceptions, rather than promote the base case
    Dinosaurs have no idea who they are protecting.

    Those exceptions ( the medical example) get used by other other folks who have less than good intentions.

  127. Steven Mosher says:

    “Mr Mosher declares that he hasn’t the faintest idea that science is not equal to open-source software engineering.”

    its pretty simple.

    If your methods are math, you share the math.
    if your methods are a standard lab proceedure, you share the fact that you used lab method X
    if your method is a survey, you share the questions,
    if your method is operating a unqiue device, say a collider, you share that fact.
    if your method is home grown code, you share the code.

    Lets take survey as an example. I surveyed americans and found out they support lower taxation
    You ask for the questions. I tell you, do your own damn survey! I tell you that it would be “better”
    if you tried to come up with your own questions rather than merely copying my questions.

    Here . dont be a dinosaur

    https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3886731/

  128. Steven Mosher says:

    dehog
    “As a thought experiment imagine Mosher dropped into the power hierarchy of Imperial Japan a day after Nagasaki. “But emperor, the Americans haven’t published their results where they are free and open for me to read them, we are not rationally obligated to even consider the science that led to the destruction of these two cities! ”

    silly boy. I may not be rationally obligated, but that doesnt prevent me from doing things for pay,
    or doing things out of mere curiosity.

    But lets suppose you come to me with results.
    you tell me the sun causes all climate change and you proved this in your recent paper.
    The paper says you ran model X. But you wont share it.

    Am I rationally obligated to consider your work? Or can I say.. you know what? if you wont
    share your method, I am not rationally obligated to regard your work.

    Now out of curiosity I may try to figure out what the hell you did or try to reconstruct what you did,
    but I would not be obligated to do so.

    Now in your example, the emporer is asking me to consider a phsyical event. Not a science claim.
    My argument is pretty simple. if you make a science claim you need to provide the data and the method. And if the method is code, share the code.

    The emporer is not asking me to asses a science CLAIM. he is asking me to explain an event.

    I’m sure you cant se the difference, because your goal isnt understanding.

  129. Steven Mosher says:

    willard;

    ‘Everything that @kmac says – and the fact that the projections and scenarios produced are not #OpenSource and so no government or other official can pull them about and understand what assumptions are or are not made.”

    I figured sooner or later folks would notice that.

    the change in the last 10 years have been huge, but watching the dinosaurs die slowly is boring

  130. Steven Mosher says:

    Let’s take an example

    https://link.springer.com/article/10.1007%2Fs13143-017-0010-y

    go through that paper an list all the calculations/adjustments/regressions that are done where
    only the results are shared and none of the actual methods.

  131. izen says:

    @-SM
    “I figured sooner or later folks would notice that.”

    It can take a while.
    https://www.ncbi.nlm.nih.gov/pubmed/27617709
    “The SRF sponsored its first CHD research project in 1965, a literature review published in the New England Journal of Medicine, which singled out fat and cholesterol as the dietary causes of CHD and downplayed evidence that sucrose consumption was also a risk factor.”

    Around 40 years in this case.
    https://www.ncbi.nlm.nih.gov/pubmed/29576024
    “PubMed search of worldwide literature was used to identify articles related to SSB and health risks published between 2001 and 2013. … CONCLUSION:
    Industry-related research during a critical period appears biased to underestimate the adverse health effects of SSB, potentially delaying corrective public health action.

    And the latest WHO recommendations in 2015 would seem to be still subject to this problem…
    https://www.who.int/bulletin/volumes/94/8/15-165852/en/
    “The draft version of the guideline included the recommendations:

    (1a) The recommendation to limit sugars intake to less than 10% of total energy is based on observational studies with dental caries as an outcome.

    (2a) The recommendation to further limit sugars intake to less than 5% of total energy is based on ecological studies in which a linear relationship between sugars intake and dental caries was observed.

    In the final version of the guideline, these were amended to (changes in italics):

    (1b) The recommendation to limit free sugars intake to less than 10% of total energy intake is based on moderate quality evidence from observational studies of dental caries.

    (2b) The recommendation to further limit free sugars intake to less than 5% of total energy intake is based on very low quality evidence from ecological studies in which a positive dose–response relationship between free sugars intake and dental caries was observed at free sugars intake of less than 5% of total energy intake.”

    What software was used is irrelevant.

  132. Steven Mosher says:

    as for medical data, dinosaurs may not be aware of what some of us are doing

    https://www.prnewswire.com/news-releases/health2sync-ctbc-bank-and-bitmark-launch-worlds-first-diabetes-data-trust-300817554.html

    when are you guys going extinct?

  133. Steven Mosher says:

    “What software was used is irrelevant.”

    You still dont get it so let me explain again.

    1. When math is your method ( say theoretical physics) you share your method.
    2. When a standard lab proceedure is your method, you share that you used method X.
    you dont for example, share the exact instruments.
    3. When your method is a survey, you share the questions.
    4. When your method is code, you share your code.

    If the code is irrelevant, then there should be no problem sharing it. Nothing lost.

    Now, dinosaurs like to share a decription of their bespoke beasts, and assume that they are sharing methods. They arent. In theoretcial physics you would not accept a paper that
    said ” I did some math” and here e=mc^2. you want to see the actual method.
    With a lab experiment, we cant share the actual device used or protocal, so we refer to
    known standards and step by step descriptions.
    with science that uses code you can ( as in math) share the ACTUAL method.
    nothing is lost by sharing the actual method.

  134. Steven Mosher says:

    izen, I do believe you win the award for silliest non sequitor, if such a thing is rateable.

    Mosher: if your method is code, you should share your method.
    izen: industry did bad science, therefore code doesnt matter.

    Did you go to the WUWT school of logic?

  135. SM wrote “You see, you guys miss the whole point. You think that having the methods ( the actual code IS the method) forces you to do something. it doesnt.”

    I know from experience that isn’t true. If you put code on-line, you will get pressure to provide support. In machine learning, that isn’t too much of a problem (except that the problem is usually at their end, and often is covered in the documentation provided), but in a contentious field like climate change, not providing support can and will be interpreted as partisan obstruction.

    The thing that puts me off providing code is the need to provide documentation so that I don’t get flooded with requests for support and advice on how to operate the software. That is a substantial additional cost.

  136. “if your method is home grown code, you share the code.”

    the code is a specific implementation of the method, at least that is most often the case in machine learning.

  137. izen says:

    @-SM
    “when are you guys going extinct?”

    Probably when the economic motivations behind research are subservient to the scientific need for transparency.

    I knew about the biases in nutritional research before climate change was a major political factor (1974?). In the analogous case of the over-consumption of an energy source that would have damaging impacts, I was not surprised to see the same pattern of manipulation of the research and how it was converted into policy, arise.

    I find the idea that public involvement in science that consists of people duplicating the maths, code or methods would correct this egregious process naive.

    I am aware of the attempts made to improve this situation, and some advances have been made. Largely voluntarily.
    However it is revealing that the recent Lancet ‘Healthy Planet Diet’ elements of which have also been incorporated in the GND, still includes a level of consumption above the best scientific recommendations.
    Do you think public involvement in the code/software methods used in climate research is going to prevent similar distortions of ‘interpretation’ and policy?

  138. izen says:

    @-SM
    ” I do believe you win the award for silliest non sequitor, if such a thing is rateable.”

    If your valuation of public involvement in science is an increase in the sale of advanced data manipulation software and a larger population that can use and understand such tools, then I accept that my point is a silly non sequitur. And your push for greater code transparency is a effective suggestion.

    If the point of increasing public involvement in science is to avoid the delay, depreciation, and denial, the distortions that shape the policy choices that arise from research, then I think your enthusiasm for an expanded coder user base is almost irrelevant.

  139. “Code is my short hand for method.”

    Then everything is fine. You just aren’t effectively communicating to scientists that weren’t raised on that lingo.

    I wrote a blog post yesterday conveying an idea so simple that it doesn’t need a detailed formulation. Could have written some software code for it, but that wouldn’t help in gaining acceptance for it. The acceptance only comes from others duplicating the idea and convincing themselves of its scientific validity.

  140. John Hartz says:

    Izen: You wrote:

    In the analogous case of the over-consumption of an energy source that would have damaging impacts, I was not surprised to see the same pattern of manipulation of the research and how it was converted into policy, arise.

    Who manipulated the research and converted it into policy?

  141. Willard says:

    > watching the dinosaurs die slowly is boring

    We may have seen nothing yet. Some newsie related to Jonathan’s work:

  142. John Hartz says:

    In addition to taking courses in how to effectively communicate to the public and policy-makers, climate scientists should also take a course or two in basic psychology and to read people.

    The problem with ignoring people’s emotions about climate change by Jeffrey T. Kiehl, Yale Climate Connections, Apr 4, 2019

  143. The Very Reverend Jebediah Hypotenuse says:


    Code is my short hand for method.

    Your API leaves something to be desired.

    Henceforth, I will assume that “purists” refers to people who use one word when they really mean a completely different word, and “data” refers to anything science-y that isn’t method.

    “Sigh” is my short-hand for: “I should know better than to read Mosher’s comments for clarity”.


    when are you guys going extinct?

    Soon enough, I suppose.

    Along with the amphibians.

  144. Joshua says:

    Oops. I meant John…

  145. “watching the dinosaurs die slowly is boring”

    I’m still here, sorry ;o)

  146. izen says:

    @-JH
    “Who manipulated the research and converted it into policy?”

    In the case of refined carbohydrates (sugars) it was the Sugar Research Foundation in the US, an industry funded body. Other similar industry allied organisations took similar roles in other nations to shape via direct funding and indirect shaping of the research field along with the long established mechanisms of regulatory capture the guidelines and policy that the WHO and various National governments then adopted. Here is a small part of the story.
    https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5099084/
    It is an example of the long evolved systems within social governance that provide the inertia to rapid or significant change when it has impacts on established economic infrastructure.

    In the case of refined hydrocarbons…
    there are several books and articles on the subject, Merchants of Doubt would be an example, but I suspect you may be better informed on that front than I.

  147. John Hartz says:

    Izen: Thanks for the clarification.

  148. John Hartz says:

    ATTP: If you did an OP about the issues raised in this article, its comment thread could very well set an all-time record for the quantity of posts.

    What If “Toxic Masculinity” Is The Reason For Climate Change? by Carolyn Centeno Milton, Forbes, Apr 3, 2019

    Introductory paragraphs:

    When a person walks out of the grocery store holding an eco-friendly canvas bag instead of a plastic bag, what gender do you think they are? Most likely, your unconscious bias answers that they are female. This is the type of answer Dr. Aaron Brough of Utah State University is trying to get to the bottom of through his research.

    Brough co-authored a paper with professors from four other universities to understand how gender norms affect sustainable decision making. They report data from seven experiments that included over 2,000 participants from the US and China. What they found was remarkable.

  149. JH,
    I’ll have a look at that.

  150. Zeeshan Amin says:

    Common people should have an understanding of basic science as they should be able to evaluate the claims of politicians whose manifestos are being increasingly stuffed with subjects like climate change.

  151. Pingback: If it seems obvious, it probably isn’t | …and Then There's Physics

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.