Where Science Usually Goes Wrong—and How Iceland Builds Better Evidence
From volcano forecasts to fisheries quotas and geothermal wells, Iceland shows why science fails when uncertainty, incentives and local knowledge are designed out of the process.
Hana BergDesign criticFirst published 10/29/2026 · monitored for updates; the next revision publishes a new version and appears here. Reader corrections are reviewed and folded into future versions.
Summary
Science rarely goes wrong because its method is useless. It goes wrong when provisional evidence is presented as certainty, measurements become detached from place, or institutions reward novelty more than correction. Iceland makes these tensions unusually visible: a volcanic forecast can alter evacuations, a fisheries model can reshape a coastal town, and a geothermal intervention can leave a physical mark on the landscape. For founders, designers and cultural innovators, the lesson is not to trust science less, but to build products, policies and narratives that expose assumptions, invite revision and respect situated knowledge.
Key takeaways
- Treat every scientific claim as a combination of evidence, model assumptions and context—not as a freestanding fact.
- A precise number can still be wrong; uncertainty ranges, base rates and measurement choices matter more than decimal places.
- Replication is essential, but field sciences such as volcanology, ecology and fisheries also require triangulation across imperfect signals.
- Conflicts of interest include more than corporate funding: publication incentives, institutional prestige and political urgency can all bend inquiry.
- Local and Indigenous knowledge should not be romanticized or dismissed; it can reveal variables, histories and failure modes that instruments overlook.
- Design is part of scientific integrity. Dashboards, maps and labels determine what decision-makers notice and misunderstand.
- The strongest innovation teams pre-register decisive tests, preserve negative results and define what evidence would make them change direction.
- Trust grows when institutions distinguish what is observed, what is inferred and what remains unknown.
Explain like I'm 5
Science is less like a book of final answers and more like a workshop for finding and fixing mistakes. Researchers observe something, propose an explanation, test it and let other people challenge the result. Trouble begins when one early answer is treated as permanent, when the test measures the wrong thing, or when nobody is rewarded for checking the work. Imagine trying to predict where lava will flow. Sensors can measure earthquakes and ground movement, but they cannot remove every surprise beneath the surface. A responsible scientist therefore offers scenarios and confidence levels, not theatrical certainty. The practical rule is simple: ask what was measured, what assumptions connect the measurement to the claim, who checked it, and what new evidence could prove it wrong.
Deep dive
The mistake is often upstream of the experiment
Popular criticism focuses on fraud, but ordinary failure is more subtle. Researchers choose which question deserves attention, which population or landscape represents the whole, what counts as an outcome, and when to stop collecting data. Those decisions shape the answer before statistical analysis begins. A laboratory study may be internally elegant yet travel poorly into a windy harbour, a remote clinic or a geothermal field. Iceland is useful here because scale makes dependencies legible: scientific institutions, municipalities, industries and communities repeatedly meet around the same fisheries, glaciers and volcanic systems. For builders, the equivalent mistake is testing a polished interface while leaving the premise untested. A mobility startup might optimize routing before asking whether residents want more traffic directed through a fragile destination. A food innovator might validate taste while ignoring energy use, imported inputs or cultural fit. Better inquiry begins with problem framing: whose problem is this, what system contains it, and which costs have been excluded?
Precision can become stagecraft
Numbers acquire authority through presentation. A single forecast, colour-coded risk map or sustainability score can conceal assumptions that specialists understand but audiences do not. During Iceland's Reykjanes Peninsula unrest, the Icelandic Meteorological Office combined seismicity, GPS deformation, satellite observations and field reports. Yet forecasts still had to contend with changing magma pathways. The honest product was not one prophecy; it was a continuously revised picture of plausible developments. This distinction matters in any evidence-led product. Observations are readings; models are structured interpretations; decisions add values and tolerances. Interfaces should separate all three. Show ranges rather than solitary numbers, timestamps rather than timeless claims, and scenarios rather than false binaries. State what the model does not include. A beautiful visualization should sharpen uncertainty, not perfume it.
Incentives distort even careful people
Modern research often rewards positive, novel and publishable findings. Small studies with flexible analyses can generate striking results, while null findings disappear into file drawers. Brian Nosek and the Open Science Collaboration highlighted the issue in 2015 when a large replication project reproduced statistically significant effects in only 36% of 100 psychology replications, compared with 97% of the original studies. That result did not invalidate psychology; it exposed a production system that had overvalued first claims. Startups repeat the pattern when experiments are designed to reassure investors. Teams selectively celebrate rising engagement, redefine success after seeing the data or run many tests until one becomes significant. The antidotes are operational: specify the hypothesis and primary metric in advance; calculate sample needs; retain failed experiments; separate exploratory analysis from confirmatory tests; and give an independent colleague permission to challenge the interpretation.
Place is evidence, not background
Standardization makes comparison possible, but it can erase the conditions that make findings true. Iceland's fisheries science demonstrates the need to connect population models, catch records, ecosystem observations and the lived knowledge of crews. Fishers may notice migration timing or changing body condition before a formal series stabilizes; scientists can test whether those observations generalize and distinguish signal from anecdote. Neither perspective is sufficient alone. The same applies to architecture and materials. A façade proven elsewhere may respond differently to Icelandic wind-driven rain, freeze-thaw cycles, salt and sparse winter light. Designers should treat local maintenance records, craft knowledge and post-occupancy feedback as data. Situated knowledge becomes rigorous when its provenance is documented, its limits are stated and it is tested against other sources.
Correction needs an interface
Self-correction is science's defining promise, but it does not happen automatically. Data need accessible documentation; code needs versioning; methods need enough detail to repeat; and institutions need graceful ways to revise advice. Retractions are conspicuous, yet quieter updates to models, classifications and guidelines are more common. A culture that treats revision as embarrassment encourages concealment. A better pattern resembles a well-maintained public object. Publish model cards, data sheets and change logs. Label preliminary findings. Explain why a recommendation changed and preserve earlier versions. Build stop rules into pilots, red-team consequential claims and invite affected communities to identify missing harms. Most importantly, define the reversal condition: what observation would cause the team to abandon its favoured explanation? Science becomes trustworthy not when it promises to be right forever, but when its pathways for becoming less wrong are visible.
Glossary
- Falsifiability
- The quality of a claim that allows evidence, at least in principle, to show it is wrong.
- Replication
- Repeating a study's methods with new data to test whether a finding appears again.
- Reproducibility
- Obtaining the same result from the same data, code and analytical procedure; terminology varies by field.
- P-hacking
- Using flexible analyses, measures or stopping rules until a statistically significant result appears.
- Publication bias
- The greater likelihood that positive or striking findings are published than null or contradictory ones.
- Pre-registration
- Time-stamping hypotheses, methods and planned analyses before observing outcomes, limiting retrospective storytelling.
- External validity
- The degree to which a finding generalizes beyond the people, place, time or conditions studied.
- Model uncertainty
- Uncertainty caused by assumptions, structure or omitted processes in a model, beyond noise in its input data.
- Triangulation
- Combining independent methods or evidence sources so that one source's weakness can be checked by another.
- Situated knowledge
- Knowledge shaped by a particular position, practice and place, valuable partly because its context is explicit.
FAQs
Does the replication crisis mean science cannot be trusted?+
No. It means individual findings deserve calibrated confidence and that some research systems need better incentives. Large, convergent bodies of evidence are generally more dependable than a single exciting paper.
How can a non-scientist judge a new claim?+
Look for sample size, comparison groups, effect size, uncertainty, pre-registration and independent replication. Also ask whether the studied setting resembles the place or population to which the claim is being applied.
Is peer review a guarantee of correctness?+
No. Peer review is a filter for plausibility, method and relevance, and reviewers can miss errors or share a field's blind spots. Publication should mark the beginning of wider scrutiny, not the end.
Why are forecasts revised so often during volcanic unrest?+
Subsurface systems cannot be observed directly in full. Agencies update interpretations as seismic, deformation, gas and field data change; revision is responsible forecasting, not necessarily evidence of incompetence.
Can local knowledge count as scientific evidence?+
It can contribute observations, hypotheses and long time horizons that formal programs lack. Its credibility improves when provenance is documented, competing explanations are considered and observations are triangulated with other evidence.
What should a startup pre-register?+
For a consequential test, record the hypothesis, eligible users, primary outcome, sample or stopping rule and analysis plan. Exploratory work can remain flexible, but it should be labelled as exploratory.
Are randomized controlled trials always the best evidence?+
They are powerful for certain causal questions, but they may be unethical, impractical or too narrow for complex systems. Natural experiments, longitudinal observations, mechanistic evidence and qualitative research can supply different forms of insight.
How should designers communicate uncertainty without confusing users?+
Use ranges, scenarios, plain-language confidence statements and visible update dates. Pair uncertainty with an action threshold: users need to know both what remains unknown and what they should do now.
Predictions
- Research interfaces will likely move from static conclusions toward living evidence objects: versioned datasets, model cards, uncertainty layers and visible correction histories.
- Iceland's volatile landscapes may accelerate digital twins that combine satellites, ground sensors and scenario modelling, although their usefulness will depend on candid limits and maintained data pipelines.
- Funders and journals may increasingly treat pre-registration, registered reports and reusable code as infrastructure rather than optional virtue.
- Product teams could adopt scientific red-teaming for sustainability and safety claims, especially as regulators scrutinize greenwashing and AI-generated evidence.
- Place-based laboratories linking scientists, craftspeople, municipalities and residents may become more valuable as climate adaptation exposes the weakness of universal solutions.
Risks
- Performative openness: teams may publish code or data that are technically public but undocumented, incomplete or impossible to reuse.
- Uncertainty laundering: probabilistic forecasts can be reduced by media, investors or officials to a single dramatic number.
- Data colonialism: institutions may extract community observations without consent, credit, governance rights or reciprocal benefit.
- Metric capture: once a proxy becomes a target—citations, engagement, catch efficiency or carbon scores—people optimize the measure while degrading the underlying goal.
- Automation bias: polished AI summaries and digital twins may invite more confidence than their source quality, spatial resolution or assumptions justify.
For professionals
For expert teams, the central design problem is epistemic governance: deciding how evidence moves from observation to inference and from inference to action. Use a claim-evidence matrix that identifies the decision, causal claim, dataset, estimator, assumptions, uncertainty class, external-validity boundary and accountable owner. Distinguish aleatory uncertainty—the variability inherent in a system—from epistemic uncertainty that further knowledge may reduce. In high-consequence settings such as volcanic hazards, marine resource management or geothermal development, combine ensemble models with decision thresholds, sensitivity analysis and documented expert elicitation. Validate not only discrimination or predictive accuracy but calibration: events assigned a 20% probability should occur roughly one time in five over an appropriate reference set. Governance should match reversibility. A low-cost, reversible prototype can tolerate weaker evidence than an intervention that alters a landscape or exposes a community to harm. Adopt registered reports or internal pre-analysis plans for pivotal claims; use multiverse analysis where reasonable analytical choices could change the conclusion; maintain lineage from raw observations to public graphic; and commission adversarial review from people outside the originating team. Qualitative evidence belongs in this architecture when collected systematically: interviews and practitioner observations can expose mechanisms, variables and distributional harms absent from aggregate metrics. The professional standard is not methodological purity. It is a traceable chain in which each method answers the question it is capable of answering—and no more.
Sources & references
- Estimating the Reproducibility of Psychological Science — Open Science Collaboration, Science (2015)
- Why Most Published Research Findings Are False — John P. A. Ioannidis, PLOS Medicine (2005)
- The Preregistration Revolution — Nosek et al., PNAS (2018)
- Reproducibility and Replicability in Science — National Academies (2019)
- The Value of Science — Richard P. Feynman (1955)
- Volcanic Activity on the Reykjanes Peninsula — Icelandic Meteorological Office
- Advice and Recommendations — Marine and Freshwater Research Institute, Iceland
- Open Science — UNESCO Recommendation and resources
| Proof theatre | Conventional validation | Open, situated inquiry | |
|---|---|---|---|
| Starting question | How can we confirm the concept? | Does the defined intervention work? | For whom, where, under which conditions—and compared with what? |
| Typical evidence | Selected testimonials and favourable metrics | Controlled test plus statistical analysis | Pre-specified tests, field evidence, qualitative insight and independent challenge |
| Treatment of place | Context is decorative | Context is controlled or averaged | Context is documented as a possible causal condition |
| Uncertainty | Hidden behind a headline number | Reported in technical intervals | Translated into ranges, scenarios, limits and decision thresholds |
| Failure record | Discarded or reframed | Sometimes retained internally | Versioned, searchable and used to update the model |
| Best fit | Pitching, with high trust risk | Bounded, repeatable questions | Complex landscapes, public-interest systems and consequential innovation |
A field guide to the studios turning cells, fungi, microbes, and living systems into cultural experiments—and potentially the next generation of materials, products, and ventures.
A design-led field guide to turning waste streams, living systems, and circular chemistry into objects people desire—and businesses built for a resource-constrained century.
Iceland is more than a dramatic backdrop. It is a compact operating system for geothermal engineering, genomic research, climate observation, circular production, and public experimentation.
Iceland’s volcanoes, genes, fisheries and energy systems reveal a larger truth: science is never only discovery. It is also a choice about ownership, risk, scale and the futures a society is willing to build.
A beginner’s field guide to reading Iceland as a living system—and turning its glaciers, volcanoes, energy networks, and cultural habits into sharper ideas for design and innovation.
Iceland shows that science is not a neat march from hypothesis to truth. It is a designed practice of observation, uncertainty, instruments, institutions, and public trust—played out on a landscape where the planet’s hidden systems rise visibly to the surface.
From our own rounds
Measured on The Curator, from real sessions people played on this site — not a third-party dataset.
- Rounds played here
- 195
- Questions per round
- 1.7