Archive for category too big to know

[2b2k] Is big data degrading the integrity of science?

Amanda Alvarez has a provocative post at GigaOm:

There’s an epidemic going on in science: experiments that no one can reproduce, studies that have to be retracted, and the emergence of a lurking data reliability iceberg. The hunger for ever more novel and high-impact results that could lead to that coveted paper in a top-tier journal like Nature or Science is not dissimilar to the clickbait headlines and obsession with pageviews we see in modern journalism.

The article’s title points especially to “dodgy data,” and the item in this list that’s by far the most interesting to me is the “data reliability iceberg,” and its tie to the rise of Big Data. Amanda writes:

…unlike in science…, in big data accuracy is not as much of an issue. As my colleague Derrick Harris points out, for big data scientists the abilty to churn through huge amounts of data very quickly is actually more important than complete accuracy. One reason for this is that they’re not dealing with, say, life-saving drug treatments, but with things like targeted advertising, where you don’t have to be 100 percent accurate. Big data scientists would rather be pointed in the right general direction faster — and course-correct as they go – than have to wait to be pointed in the exact right direction. This kind of error-tolerance has insidiously crept into science, too.

But, the rest of the article contains no evidence that the last sentence’s claim is true because of the rise of Big Data. In fact, even if we accept that science is facing a crisis of reliability, the article doesn’t pin this on an “iceberg” of bad data. Rather, it seems to be a melange of bad data, faulty software, unreliable equipment, poor methodology, undue haste, and o’erweening ambition.

The last part of the article draws some of the heat out of the initial paragraphs. For example: “Some see the phenomenon not as an epidemic but as a rash, a sign that the research ecosystem is getting healthier and more transparent.” It makes the headline and the first part seem a bit overstated — not unusual for a blog post (not that I would ever do such a thing!) but at best ironic given this post’s topic.

I remain interested in Amanda’s hypothesis. Is science getting sloppier with data?

Tags:

Elsevier acquires Mendeley + all the data about what you read, share, and highlight

I liked the Mendeley guys. Their product is terrific — read your scientific articles, annotate them, be guided by the reading behaviors of millions of other people. I’d met with them several times over the years about whether our LibraryCloud project (still very active but undergoing revisions) could get access to the incredibly rich metadata Mendeley gathers. I also appreciated Mendeley’s internal conflict about the urge to openness and the need to run a business. They were making reasonable decisions, I thought. At they very least they felt bad about the tension :)

Thus I was deeply disappointed by their acquisition by Elsevier. We could have a fun contest to come up with the company we would least trust with detailed data about what we’re reading and what we’re attending to in what we’re reading, and maybe Elsevier wouldn’t win. But Elsevier would be up there. The idea of my reading behaviors adding economic value to a company making huge profits by locking scholarship behind increasingly expensive paywalls is, in a word, repugnant.

In tweets back and forth with Mendeley’s William Gunn [twitter: mrgunn], he assures us that Mendeley won’t become “evil” so long as he is there. I do not doubt Bill’s intentions. But there is no more perilous position than standing between Elsevier and profits.

I seriously have no interest in judging the Mendeley folks. I still like them, and who am I to judge? If someone offered me $45M (the minimum estimate that I’ve seen) for a company I built from nothing, and especially if the acquiring company assured me that it would preserve the values of that company, I might well take the money. My judgment is actually on myself. My faith in the ability of well-intentioned private companies to withstand the brute force of money has been shaken. After all this time, I was foolish to have believed otherwise.

MrGunn tweets: “We don’t expect you to be joyous, just to give us a chance to show you what we can do.” Fair enough. I would be thrilled to be wrong. Unfortunately, the real question is not what Mendeley will do, but what Elsevier will do. And in that I have much less faith.

 


I’ve been getting the Twitter handles of Mendeley and Elsevier wrong. Ack. The right ones: @Mendeley_com and @ElsevierScience. Sorry!

Tags:

[annotation][2b2k] Critique^it

Ashley Bradford of Critique-It describes his company’s way of keeping review and feedback engaging.

NOTE: Live-blogging. Getting things wrong. Missing points. Omitting key information. Introducing artificial choppiness. Over-emphasizing small matters. Paraphrasing badly. Not running a spellpchecker. Mangling other people’s ideas and words. You are warned, people.

To what extent can and should we allow classroom feedback to be available in the public sphere? The classroom is a type of Habermasian civic society. Owning one’s discourse in that environment is critical. It has to feel human if students are to learn.

So, you can embed text, audio, and video feedback in documents, video and images. It translates docs into HTML. To make the feedback feel human, it uses slightly stamps. You can also type in comments, marking them as neutral, positive, or critique. A “critique panel” follows you through the doc as you read it, so you don’t have to scroll around. It rolls up comments and stats for the student or the faculty.

It works the same in different doc types, including Powerpoint, images, and video.

Critiques can be shared among groups. Groups can be arbitrarily defined.

It uses HTML 5. It’s written in Javascript, PHP, and uses Mysql.

“We’re starting with an environment. We’re building out tools.” Ashley aims for Critique^It to feel very human.

Tags:

[annotation][2b2k] Mediathread

Jonah Bossewich and Mark Philipsonfrom Columbia University talk about Mediathread, an open source project that makes it easy to annotate various digital sources. It’s used in many courses at Columbi, as well as around the world.

NOTE: Live-blogging. Getting things wrong. Missing points. Omitting key information. Introducing artificial choppiness. Over-emphasizing small matters. Paraphrasing badly. Not running a spellpchecker. Mangling other people’s ideas and words. You are warned, people.

It comes from Columbia’s Center for New Media Teaching and Learning. It began with Vital, a video library tool. It let students clip and save portions of videos, and comment on them. Mediathread connects annotations to sources by bookmarking, via a bookmarklet that interoperates with a variety of collections. The bookmarklet scrapes the metadata because “We couldn’t wait for the standards to be developed.” Once an item is in Mediathread, it embeds the metadata as well.

It has always been conceived of a “small-group sharing and collaboration space.” It’s designed for classes. You can only see the annotations by people in your class. It does item-level annotation, as well as regions.

Mediathread connects assignments and responses, as well as other workflows. [He's talking quickly :)]

Mediathread’s bookmarklet approach requires it to have to accommodate the particularities of sites. They are aiming at making the annotations interoperable in standard forms.

Tags:

[annotation][2b2k] Paolo Ciccarese on the Domeo annotation platform

Paolo Ciccarese begins by reminding us just how vast the scientific literature is. We can’t possibly read everything we should. But “science is social” so we rely on each other, and build on each other’s work. “Everything we do now is connected.”

NOTE: Live-blogging. Getting things wrong. Missing points. Omitting key information. Introducing artificial choppiness. Over-emphasizing small matters. Paraphrasing badly. Not running a spellpchecker. Mangling other people’s ideas and words. You are warned, people.

Today’s media do provide links, but not enough. Things are so deeply linked. “How do we keep track of it?” How do we communicate with others so that when they read the same paper they get a little bit of our mental model, and see why we found the article interesting?

Paolo’s project — Domeo [twitter:DomeoTool] — is a web app for “producing, browsing, and sharing manual and semi-automatic (structure and unstructured) annotations, using open standards. Domeo shows you an article and lets you annotate fragments. You can attach a tag or an unstructured comment. The tag can be defined by the user or by a defined ontology. Domeo doesn’t care which ontologies you use, which means you could use it for annotating recipes as well as science articles.

Domeo also enables discussions; it has a threaded msg facility. You can also run text mining and entity recognition systems (Calais, etc.) that automatically annotates the work with those words, which helps with search, understanding, and curation. This too can be a social process. Domeo lets you keep the annotation private or share it with colleagues, groups, communities, or the Web. Also, Domeo can be extended. In one example, it produces information about experiments that can be put into a database where it can be searched and linked up with other experiments and articles. Another example: “hypothesis management” lets readers add metadata to pick out the assertions and the evidence. (It uses RDF) You can visualize the network of knowledge.

It supports open APIs for integrating with other systems., including into the Neuroscience Information Framework and Drupal. “Domeo is a platform.” It aims at supporting rich source, and will add the ability to follow authors and topics, etc., and enabling mashups.

Tags:

[annotation][2b2k] Neel Smith: Scholarly annotation + Homer

Neel Smith of Holy Cross is talking about the Homer Multitext project, a “long term project to represent the transmission of the Iliad in digital form.”

NOTE: Live-blogging. Getting things wrong. Missing points. Omitting key information. Introducing artificial choppiness. Over-emphasizing small matters. Paraphrasing badly. Not running a spellpchecker. Mangling other people’s ideas and words. You are warned, people.

He shows the oldest extant ms of the Iliad, which includes 10th century notes. “The medieval scribes create a wonderful hypermedia” work.

“Scholarly annotation starts with citation.” He says we have a good standard: URNs, which can point to, for example, and ISBN number. His project uses URNs to refer to texts in a FRBR-like hierarchy [works at various levels of abstraction]. These are semantically rich and machine-actionable. You can google URN and get the object. You can put a URN into a URL for direct Web access. You can embed an image into a Web page via its URN [using a service, I believe].

An annotation is an association. In a scholarly notation, it’s associated with a citable entity. [He shows some great examples of the possibilities of cross linking and associating.]

The metadata is expressed as RDF triples. Within the Homer project, they’re inductively building up a schema of the complete graph [network of connections]. For end users, this means you can see everything associated with a particular URN. Building a facsimile browser, for example, becomes straightforward, mainly requiring the application of XSL and CSS to style it.

Another example: Mise en page: automated layout analysis. This in-progress project analyzes the layout of annotation info on the Homeric pages.

Tags:

[annotations][2b2k] Rob Sanderson on annotating digitized medieval manuscripts

Rob Sanderson [twitter:@azaroth42] of Los Alamos is talking about annotating Medieval manuscripts.

NOTE: Live-blogging. Getting things wrong. Missing points. Omitting key information. Introducing artificial choppiness. Over-emphasizing small matters. Paraphrasing badly. Not running a spellpchecker. Mangling other people’s ideas and words. You are warned, people.

He says many Medieval manuscripts are being digitized. The Mellon Foundation is funding many such projects. But these have tended to reinvent the same tech, and have not been designed for interoperability with other projects. So the Digital Medieval Initiative was founded, with a long list of prestigious partners. They thought about what they’d like: distributed, linked data, interoperable, etc. For this they need a shared description format.

The traditional approach is annotate an image of a page. But it can be very difficult to know which images to annotate; he gives as an example a page that has fold-outs. “The naive assuption is that an image equals a page.” But there may be fragments, or only portions of the page have been digitized (e.g., the illuminations), etc. There may be multiple images on a page, revealed by multi-spectral imaging. There may be multiple orientations of the page, etc.

The solution? The canvas paradigm. A canvas is an empty space corresponding to the rectangle (or whatever) of the page. You allow rich resources to be associated with it, and allow users to comment. For this, they use Open Annotation. You can specify a choice of images. You can associate text with an area of the canvas. There are lots of different ways to visualize those comments: overlays, side-by-side, etc.

You can build hybrid pages. For example, and old scan might have a new color scan of its illustrations pointing at it. Or you could have a recorded performance of a piece of music pointing at the musical notation.

In summary, the SharedCanvas model uses open standards (HTML 5, Open Annotation, TEI, etc.) and can be implement distributed across reporsitories, encouraging engagement by domain experts.

Tags:

[2b2] Data, facts, and the comfort of decisions

Just a quick note updating my post yesterday about the musky Tesla-Times affair. [('m in an airport with just a few minutes before boarding.)

Times Man John Broder has posted his step-by-step rebuttal-explanation-apologia of Elon Musk's data-driven accusations that Broder purposefully drove a Tesla S into a full stop. Looked at purely as a drama of argument, it just gets more and more fascinating. But it is of course not merely a drama or an example; reputations of people are at stake, and reputations determine careers and livelihoods.

Broder's overall defense is that he was on the phone with Tesla support at most of the turning points, and followed instructions scrupulously. As a result, just about every dimension of this story is now in play and in question: Were the data accurate or did Broder misremember turning on cruise control? Were the initial conditions accounted for (e.g., different size wheels)? Were the calculations based on that data accurate, or are the Tesla algorithms off when the weather is cold? Does being a first-time driver count as a normal instance? Does being 100% reliant on the judgment of support technicians make a test optimal or atypical? Should Broder have relied on what the instruments in the car said or what Support told him? If a charging pump is in a service area but no one sees it, does it exist?

And then there's the next level. We humans live with this sort of uncertainty — multi-certainty? — all the time. It's mainly what we talk about when given a chance. For most of us, it's idle chatter — you get to rail against the NY Times, I get to write about data and knowledge, and Tesla car owners get to pronounce in high dudgeon. Fun for all. But John Broder's boss is going to have to decide how to respond. It's quite likely that that decision is going to reflect the murky epistemology of the situation. Evidence will be weighed and announced to be probabilistic. Policy guidelines will be consulted. Ultimately the decision is likely to be pegged to a single point of policy, phrased as something like, "In order to maintain the NYT's reputation against even unlikely accusations, we have decided to ..." or "Because our reviewer followed every instruction given him by Tesla..." Or some such; I'm not trying to predict the actual decision, but only that it will prioritize one principle from among dozens of possibilities.

Thus, as is usually the case, the decision will force a false sense of closure. It will pick one principle, and over time, the decision will push an even grosser simplification, for people will remember which way the bit flipped — fired, suspended, backed fully, whatever — but not the principle, not the doubt, not the unredeemable uncertainty. This case will become yet one more example of something simple &mdash masking the fathomless complexity revealed even by a single review of a car.

That complexity is now permanently captured in the web of blue underlined text. We can always revisit it. But, we won't, because the matter was decided, and decisions betray complexity.

[Damn. Wish I had time to re-read this before posting! Forgive typos, thinkos, etc.?]

Tags:

[2b2k] Are all good conversations echo chambers?

Bora Zivkovic, the blog editor at Scientific American, has a great post about bad comment threads. This is a topic that has come up every day this week, which may just be a coincidence, or perhaps is a sign that the Zeitgeist is recognizing that when it talks to itself, it sounds like an idiot.

Bora cites a not-yet-published paper that presents evidence that a nasty, polarized comment thread can cause readers who arrive with no opinion about the paper’s topic to come to highly polarized opinions about it. This is in line with off-line research Cass Sunstein cites that suggests echo chambers increase polarization, except this new research indicates that it increases polarization even on first acquaintance. (Bora considers the echo chamber idea to be busted, citing a prior post that is closely aligned with the sort of arguments I’ve been making, although I am more worried about the effects of homophily — our tendency to hang out with people who agree with us — than he is.)

Much of Bora’s post is a thoughtful yet strongly voiced argument that it is the responsibility of the blog owner to facilitate good discussions by moderating comments. He writes:

So, if I write about a wonderful dinner I had last night, and somewhere in there mention that one of the ingredients was a GMO product, but hey, it was tasty, then a comment blasting GMOs is trolling.

Really? Then why did Bora go out of his way to mention that it was a GMO product? He seems to me to be trolling for a response. Now, I think Bora just picked a bad example in this case, but it does show that the concept of “off-topic” contains a boatload of norms and assumptions. And Bora should be fine with this, since his piece begins by encouraging bloggers to claim their conversation space as their own, rather than treating it as a public space governed by the First Amendment. It’s up to the blogger to do what’s necessary to enable the type of conversations that the blogger wants. All of which I agree with.

Nevertheless, Bora’s particular concept of being on-topic highlights a perpetual problem of conversation and knowledge. He makes a very strong case — nicely argued — for why he nukes climate-change denials from his comment thread. Read his post, but the boiled down version is: (a) These comments are without worth because they do not cite real evidence and most of them are astroturf anyway. (b) They create a polarized environment that has the bad effect of raising unjustified doubts in the minds of readers of the post (as per the research he mentions at the beginning of his post). (c) They prevent conversation from advancing thought because they stall the conversation at first principles. Sounds right to me. And I agree with his subsequent denial of the echo chamber effect as well:

The commenting threads are not a place to showcase the whole spectrum of opinions, no matter how outrageous some of them are, but to educate your readers, and to, in turn, get educated by your readers who always know something you don’t.

But this is why the echo chamber idea is so slippery. Conversation consists of the iteration of small differences upon a vast ground of agreement. A discussion of a scientific topic among readers of Scientific American has value insofar as they can assume that, say, evolution is an established theory, that assertions need to be backed by facts of a certain evidentiary sort (e.g., “God told me” doesn’t count), that some assertions are outside of the scope of discussion (“Evolution is good/evil”), etc. These are criteria of a successful conversation, but they are also the marks of an echo chamber. The good Scientific American conversation that Bora curates looks like an echo chamber to the climate change deniers and the creationists. If one looks only at the structure of the conversation, disregarding all the content and norms, the two conversations are indistinguishable.

But now I have to be really clear about what I’m not saying. I am not saying that there’s no difference between creationists and evolutionary biologists, or that they are equally true. I am not saying that both conversations follow the same rules of evidence. I am certainly not saying that their rules of evidence are equally likely to lead to scientific truths. I am not even saying that Bora needs to throw open the doors of his comments. I’m saying something much more modest than that: To each side, the other’s conversation looks like a bunch of people who are reinforcing one another in their wrong beliefs by repeating those beliefs as if they were obviously right. Even the conversation I deeply believe is furthering our understanding — the evolutionary biologists, if you haven’t guessed where I stand on this issue — has the structure of an echo chamber.

This seems to me to have two implications.

First, it should keep us alert to the issue that Bora’s post tries to resolve. He encourages us to exclude views challenging settled science because including ignorant trolls leads casual visitors to think that the issues discussed are still in play. But climate change denial and creationist sites also want to promote good conversations (by their lights), and thus Bora is apparently recommending that those sites also should exclude those who are challenging the settled beliefs that form the enabling ground of conversation — even though in this case it would mean removing comments from all those science-y folks who keep “trolling” them. It seems to me that this leads to a polarized culture in which the echo chamber problem gets worse. Now, I continue to believe that Bora is basically right in his recommendation. I just am not as happy about it as he seems to be. Perhaps Bora is in practice agreeing with Too Big to Know’s recommendation that we recognize that knowledge is fragmented and is not going to bring us all together.

Second, the fact that we cannot structurally distinguish a good conversation from a bad echo chamber I think indicates that we don’t have a good theory of conversation. The echo chamber fear grows in the space that a theory of conversation should inhabit.

I don’t have a theory of conversation in my hip pocket to give you. But I presume that such a theory would include the notion, evident in Bora’s post, that conversations have aims, and that when a conversation is open to the entire world (a radically new phenomenon…thank you WWW!) those aims should be explicitly stated. Likewise for the norms of the conversation. I’m also pretty sure that conversations are never only about they say they’re about because they are always embedded in complex social environments. And because conversations iterate on differences on a vast ground of similarity, conversations rarely are about changing people’s minds about those grounds. Also, I personally would be suspicious of any theory of conversation that began by viewing conversations as composed fundamentally of messages that are encoded by the sender and decoded by the recipient; that is, I’m not at all convinced that we can get a theory of conversation out of an information-based theory of communication.

But I dunno. I’m confused by this entire topic. Nothing that a good conversation wouldn’t cure.

Tags:

[2b2k] Redditopedia

Reddit user (or, as they say, redditor) thefuc has posted 100+ links to Reddit discussions of questions about Roman history. Samples include:

How “sudden” was the fall of Rome?

How long did the worship of Roman gods continue after the fall of the Roman empire?

What set Byzantium apart technologically or culturally, and why did poets like W.B. Yeats go on to romanticize it?

How did different “barbaric” groups perceive the Romans?

What are the current views on whether there was a post-Roman mass-migration of Anglo-Saxons to Britain during the dark ages or whether it was mainly a cultural and technological change with the ruling classes most affected by warrior migration?

Did the Romans simply leave Briton with the collapse of their empire

What happened to urban planning after the fall of Rome?

Why do all helmets seem to loose their cheek pieces during the 5-10th centuries?

I haven’t read these,; but it’s a really interesting model, especially if the threads are good. (If they’re not, they’re an interesting model of something else.) It indicates one way knowledge works at scale: Thousands of questions and discussions, and a culling that addresses both some of the basics but also many of the quirks. Pretty damn fascinating.

Tags: