How the score is calculated
Every paper carries three separate judgements. Rigour asks can I believe this?, the clinical score asks if it holds up, how far should it move patient care?, and editorial priority asks why should I read it this week?
merit = [clinical + 0.5(editorial priority − 45)] × credibility(rigour) × synthesis certainty
heat = merit − excess missingness uncertainty − recency
credibility(r) = 0.35 + 0.65 (r/100)0.7
The clinical score anchors the rank. Editorial priority contributes only its deviation from a neutral 45, so novelty can change reading order without inflating the estimated clinical effect. The floor of 0.35 means weak methods discount a finding rather than annihilating it, and credibility reaches exactly 1.00 at rigour 100 so nothing is ever inflated. The recency term takes off 1.5 points a week after the first week, capped at 12; it orders the live feed only, and the band a paper displays is taken before it, so nothing quietly drops a band three weeks after publication and an archived paper reads the same as it did on the day. A retraction zeroes all four numbers.
Three rules shape the rigour assessment. Design sets a starting point and a ceiling, not a verdict. Gates fire only on observed defects, never on absence. Signals add up inside a domain, and domains cap rather than sum. The last is the shape of RoB 2, ROBINS-I and AMSTAR-2, and it is the answer to the 1999 demonstration that additive quality scales give contradictory answers about the same trials.
Drive it yourself
This runs the actual scoring module, not a simplified copy of it. Set a design and an appraisal and watch the numbers move. Change patient consequence, decision relevance or applicability. Editing a confidence interval changes how clearly the paper answered its question (whether the interval excludes no difference) but never the relevance points, and the size of the effect earns nothing. The examples are illustrative appraisals, not a validation dataset.
The paper
How it was done
Clinical
Results and editorial context
- 95
- 86
- 54
merit starts with clinical 86 + 0.5 × (editorial 54 − 45), then applies credibility(95) 0.977
coverage 1.00 · heuristic half-width ±6 · rank debit 0
Mean difference 0.75 (95% CI -1.33 to 2.84; Oxford Shoulder Score). The interval includes no difference. This does not establish equivalence or rule out an important effect.
- Patient consequence
- 3/3
- Decision relevance
- 3/3
- Applicability
- 3/3
How each method changes rigour
| Criterion | Yes | Silent | No |
|---|---|---|---|
| Allocation concealment | 95 | 90 | 81 |
| Blinded outcome assessment | 95 | 88 | 79 |
| Random sequence generation | 95 | 93 | 86 |
| Intention-to-treat analysis | 95 | 92 | 86 |
| Prospective registration | 95 | 90 | 90 |
| Aligned time zero | 95 | 95 | 95 |
| Competing risks handled | 99 | 95 | 95 |
| Harms reported | 95 | 95 | 95 |
The widget exposes the criteria that move a score most, not all appraisal fields. It runs the same scoring function as the weekly pipeline. Confidence intervals stay in their original reported units.
Rigour, in full
Rigour starts from a design prior drawn from a 47-family table and is held under a ceiling from the same row. The starting points are compressed and the ceilings wide, so execution decides where in the corridor a paper lands. A sham-controlled trial starts at 68.6 with no ceiling below 100; an adjusted registry cohort starts at 37.4 with a ceiling of 80 and therefore has 42.6 points of headroom; a retrospective case series starts at 8.8 under a ceiling of 42.
The ceiling is structural rather than a judgement about quality. A cross-sectional study cannot establish temporality however well it is conducted, and a cadaveric construct cannot speak to patient outcomes. Preclinical families are exempt from the spread that separates the clinical hierarchy, because their limitation belongs to the clinical score, whose design ceiling holds them low, not to rigour: good bench science should read as good science answering a different question.
Each credited criterion then adds or subtracts exactly what the tables below say, scaled once by a constant 0.85. There is no variable discounting: an earlier build shrank the sum toward the prior as the appraisal thinned, and that multiplier could make an observed defect raise the score, so it was removed. A thin appraisal now sits near its prior simply because few criteria moved it. Sparseness widens the uncertainty band and subtracts the excess uncertainty from the weekly rank; a paper appraised on under 60% of its applicable criteria cannot lead the week. Journal tier is the only proxy left in the model, worth at most 6 points, and it decays to exactly zero once 80% of the applicable criteria have been read.
A property test now holds the whole axis to that contract: observing a strength can only raise a score, observing a defect can only lower it, and silence moves nothing.
| Concealed allocation | +6 |
| Allocation not concealedon a subjective outcome; −5 on an objective one; −8 when the outcome's objectivity was not reported, the midpoint rather than the worst case | −11 |
| Random sequence generation | +3 |
| Quasi-random allocation | −8 |
| Sham-controlleda bonus, and does not enlarge the coverage denominator | +8 |
| Adjustment-method labelmatching, regression, propensity scores, IPTW, g-methods and IV are neutral by themselves; sophistication is not adequacy | 0 |
| Unadjusted causal comparisona significant effectiveness claim also caps rigour at 50 | −8 to −10 |
| Post-adjustment balance demonstratedfor matching, propensity scores or weighting; important residual imbalance −6 | +5 |
| New-user designfor an intervention comparison; including prevalent users −4 | +4 |
| Exposure defined after cohort entryimmortal time bias, and it caps rigour at 25 | −14 |
| Comparatorplacebo or sham +6, current standard +4, accepted alternative +2, no treatment 0, outdated −10; scored only where a comparison group was affirmed | −10 to +6 |
| Blinded outcome assessmenton a subjective outcome; +4 on an objective one; +6 when objectivity was not reported | +8 |
| Unblinded assessment of a subjective outcomethe largest single criterion penalty in the model; −4 objective, −7.5 unreported | −11 |
| Registry coding validated | +3 |
| Independently adjudicated outcomes | +4 |
| Registry capturea continuous ramp from nothing at 85% capture to the full credit at 98% | 0 to +5 |
Blinding carries the heaviest penalty because the evidence says it should. Unblinded assessors exaggerate odds ratios by about 36% and standardised mean differences by 68%, and in orthopaedics specifically the gap is a standardised mean difference of 0.76 unblinded against 0.25 blinded.
| Intention-to-treat analysis | +4 |
| Per-protocol analysis only | −7 |
| Follow-up short of the window the question needsa continuous ramp, largest when nothing of the window was observed. Below half the window a cap also engages, relaxing linearly as the shortfall closes. Arthroplasty survivorship needs 60 months; fracture union 9; periprosthetic infection 24 | 0 to −12 |
| Follow-up well past the thresholdramping up toward the generous benchmark for the question | 0 to +4 |
| Loss to follow-upa continuous ramp from +2 at no loss to −10 at 35%. The old bins charged 7 points for one patient crossing 20% | +2 to −10 |
| Crossover between armsramping down to −14 at 30% crossover; at 40% or more, rigour is capped at 65, because the arms as analysed are no longer the arms as randomised. Operative-versus-conservative crossover routinely exceeds 20% in orthopaedic trials and dilutes the estimate toward the null | +2 to −14 |
| Adequately powered | +7 |
| Underpowered | −10 |
| No power calculation reportednot scored at all: it holds on 78% of the corpus, so it is an offset rather than a discriminator | 0 |
| Prospectively registereda bonus and never a penalty, because only about a quarter of orthopaedic surgical trials report it | +6 |
| Headline is not the primary outcome | −8 |
| No correction for multiple comparisons | −4 |
| A positive result from under 50 patientsunder 100, −4 | −8 |
| A positive result from a single centre | −5 |
| An unpowered null from under 50 patientsunder 150, −4. A different defect from exaggeration: this debits an uninformative null for being uninformative, and never touches a null that was adequately powered | −7 |
| A single centre's own patients, in observational workselection, not exaggeration, so it applies to cohorts and series whichever way the result went | −4 |
| Cases selected rather than consecutiveconsecutive enrolment earns +3 | −6 |
| Fragility index at or below the subspecialty medianabove it +1, at three times it +3. Medians: arthroplasty 1, sports and spine 2, trauma 3, foot and ankle 6 | −5 |
| More patients lost than the result can absorb | −8 |
| Reverse fragility index of 5 or morefires only on a non-significant result, where a high value means the null is robust | +4 |
The size and single-centre penalties on positive results fire only when the primary outcome was significant, because small-study and single-centre inflation are directional: they exaggerate positive findings and cannot manufacture a null.
| Pools randomised trialsmixed designs −4, observational −8, uncontrolled case series −14 | +6 |
| I² and reported GRADE certaintyneither substitutes for review conduct: certainty discounts the rank, and a review needs moderate or high certainty before the clinical score treats its answer as settled | 0 conduct points |
| Comprehensive literature search | +4 |
| Risk of bias assessed in the included studiesand carried into the synthesis rather than parked in a table, +3 more | +4 |
| Excluded studies listed and justified | +2 |
| Publication bias assessedaffirmatively NOT assessed across ten or more pooled studies, −8 | +3 |
| Two or more AMSTAR-2 critical failuresone caps at 64. Checked domains are protocol registration, search, excluded-study justification, risk-of-bias assessment and interpretation, plus publication-bias assessment when enough studies were pooled | cap 42 |
A synthesis is no longer charged criteria it cannot satisfy. An earlier build expected assessor blinding, a follow-up window and a power calculation of every meta-analysis, none of which a meta-analysis has, and the missing denominators quietly barred the one design that can close a question from ever leading the week.
| Immortal time bias | 25 |
| A bench study drawing a clinical conclusion | 30 |
| Data leakage between training and test sets | 30 |
| Two-gate sampling, conclusions the results do not support, or a prediction model with no external validation | 40 |
| A pilot making an efficacy claim | 45 |
| An effectiveness claim with no adjustment for confounding | 50 |
| A conclusion asserting a benefit the primary outcome did not shownever applied to a non-inferiority or equivalence trial, where concluding the treatments are comparable is correct reporting rather than spin | 55 |
| The outcome window was not reachedrelaxing linearly as the shortfall closes, so a month of follow-up cannot move the score 33 points | 60 up |
| Crossover of 40% or more between arms | 65 |
The clinical score, in full
From scorer version 12, Clinical asks how far a paper should move patient care if its methods hold up. It multiplies three things: the most its design can claim, how relevant its question is, and how clearly it answered that question. Earlier published scores keep the calculation they were made with and are identified in each appraisal.
clinical = design ceiling × (relevance points ÷ 9) × answer clarity
This is an editorial rubric, not a validated clinical scale. Its weights are explicit policy choices and need independent clinician calibration. No MCID library enters it, and the size of an effect earns nothing.
| Patient consequenceHow much does the PRIMARY OUTCOME this study actually measured matter to patients? Judge the endpoint and its time point, not how serious the condition is. | 0–3 |
| Decision relevanceIf this result is true, which specific choice would a clinician make differently, and how directly does THIS study's primary result bear on that choice? | 0–3 |
| ApplicabilityHow closely do the patients, the setting and the way the options were delivered match the people and the care where this result would be used? | 0–3 |
Patient consequence: how the levels differ
- 3 · High
- The primary outcome is itself a major, durable patient outcome: death, limb loss, permanent disability, reoperation or revision, a serious complication, or validated patient-reported function or pain at a time point that matters for the decision (usually a year or more after elective surgery). Reserve for clearly major outcomes; most papers do not qualify.
- 2 · Moderate
- A patient-experienced outcome of real but smaller or shorter-lived importance: early pain or function, speed of recovery, return to sport or work, minor complications, satisfaction, or a validated score at a short time point.
- 1 · Limited
- A surrogate or intermediate outcome whose link to how patients feel or function is uncertain: radiographic or imaging findings, laboratory values, biomechanical or range-of-motion measures, healing on imaging, process or resource measures; or a trivial patient outcome.
- 0 · None
- No patient outcome is measured or implied.
Decision relevance: how the levels differ
- 3 · High
- The study compares the actual options a clinician chooses between (or a whole strategy), and its primary result directly indicates which to choose, or that they are interchangeable. A clinician could act on it without a further translation step.
- 2 · Moderate
- Bears on a real decision, but indirectly: an association or risk factor that informs counselling or surveillance, a subgroup or secondary analysis, one component of a pathway, or a comparison that leaves out a main alternative.
- 1 · Limited
- Descriptive, exploratory, mechanistic or hypothesis-generating; no specific clinical choice would change on this result alone.
- 0 · None
- No clinical decision is identifiable.
Applicability: how the levels differ
- 3 · High
- Typical patients for this decision, in the kind of setting where the decision is made (routine or specialist care), with the options delivered as they would be in practice. Rare and specialist conditions qualify when this holds. Multicentre recruitment alone does not make a study high.
- 3 · Moderate
- A recognisable practice population, but with a notable restriction: narrow eligibility, a single expert surgeon or centre, or an unusual protocol that would need adapting.
- 1.5 · Limited
- Highly selected patients, experimental delivery, or a setting far from routine care.
- 0 · None
- No current application to patient care.
Each judgement must be supported by a quotation found in the paper. Patient consequence and decision relevance score 0 to 3. Applicability only subtracts: high and moderate both count in full, limited counts half and none counts nothing, because nearly every study runs under some restriction and that alone should not cost points. Patient consequence is held at limited when the primary outcome is a radiographic, physiological, laboratory or biomechanical measure, except for diagnostic questions, where the measurement is the point. If any judgement cannot be made, Clinical is shown as incomplete rather than zero, and the paper goes to review.
| Clear answerThe primary interval excludes no difference, in either direction (for a review, at moderate or high GRADE certainty), or a prespecified margin was clearly failed. | × 1.00 |
| Clear answer of no important differenceNo difference from a trial powered to find one, or from a review rating its certainty moderate or high, or a prespecified non-inferiority or equivalence margin met. | × 0.94 |
| Points one wayA verdict without a usable interval, a clear result from a trial the paper itself calls underpowered, or a review whose certainty is low or unrated. | × 0.86 |
| No difference found, precision unclearNo difference was found, but nothing shows the study could have found one: no power calculation, or a review whose certainty is low or unrated. Treated as neutral. | × 0.80 |
| Result not capturedNo primary interval or verdict was extracted. Treated as neutral: neither credited as an answer nor charged as inconclusive. | × 0.80 |
| InconclusiveNo difference from a trial the paper itself calls underpowered, or a prespecified margin was crossed. | × 0.70 |
| Descriptive by designThe design makes no comparison of patient outcomes (a descriptive or laboratory study), so its ceiling alone limits the score. | × 1.00 |
A clear harm is as much an answer as a clear benefit. A finding of no difference counts as an answer only against a yardstick the paper set itself: a power calculation for the difference it set out to find, a prespecified non-inferiority or equivalence margin, or a review's own GRADE certainty. Silence about that yardstick is neutral. Where the extracted result leaves the answer open (nothing captured, or no difference with no power calculation or certainty rating), the appraisal's own quoted judgement of how clearly the paper answered its question settles it, under the same rules: a review still needs moderate or high certainty, a trial the paper calls underpowered still only points one way, and an extracted no-difference can never be judged into a difference. A demonstrated difference edges out an otherwise identical decisive null by a slim margin, and both clear an inconclusive result by far more.
| IPD meta-analysis of RCTs | 100 |
| Meta-analysis of RCTs | 98 |
| Network meta-analysis | 94 |
| Multicentre RCT | 92 |
| Sham-controlled RCT | 92 |
| Systematic review of RCTs | 92 |
| Single-centre RCT | 85 |
| Meta-analysis (mixed designs) | 82 |
| Target trial emulation | 78 |
| Prospective cohort | 72 |
| Meta-analysis of observational studies | 70 |
| Inception cohort | 68 |
| Retrospective cohort | 65 |
| Diagnostic accuracy | 62 |
| Cross-sectional | 55 |
| Retrospective case series | 45 |
| Biomechanical / cadaveric | 40 |
| Narrative review | 25 |
The top of the scale is reserved for a finding that changes orthopaedic surgery, one that overturns a guideline or becomes one. A single trial is one concordant study, and a strong guideline recommendation needs two or more, so no single primary study can score above 92; the headroom above that belongs to syntheses of randomised trials. Observational designs stop lower again, because on their own they cannot close a treatment question. The ceiling says how far this kind of evidence can move practice; how well the study was run is rigour's job.
Effect estimates and confidence intervals are shown in their reported units. A paper-reported MCID may accompany them as source-verified context; it never changes a score. A confidence interval including no difference is not proof of equivalence.
Non-inferiority and equivalence retain their reported margins. Interpretation distinguishes an interval within the margin, one crossing a boundary, and one entirely beyond it. It does not infer that a margin is clinically acceptable merely because it is smaller than an MCID. Margin justification and the trial's analysis assumptions still require appraisal. Missing margins produce no verdict, and a formal superiority claim requires the appropriate analysis.
| Contradicts itStrong +20, Moderate +18, Limited +14, Consensus +12 | +12 to +20 |
| Fills a gap it statespays most where the guideline concedes the evidence is limited | +8 to +18 |
| Confirms itconfirming a weak recommendation is worth more than confirming a strong one, because a Strong recommendation needs two or more concordant high-quality studies. That row is what stops the model punishing replication | +2 to +10 |
| A relation we cannot tie to a real recommendationit is then a claim about the literature, not a collision with guidance | × 0.4 |
| Prior evidenceFirst evidence +6; resolves conflicting trials +5; first randomised evidence +4 (randomised trials only); adds to consistent evidence −2. | −2 to +6 |
| Guideline relation and novelty togetherThese partly overlap, so their combined contribution is limited. | at most +24 |
Spin and reporting completeness do not touch the clinical score. Methods and unsupported conclusions affect rigour and ranking. The leading rank band requires a complete clinical assessment and at least 60% appraisal coverage.
How to read the bands
The five bands are exceptional from 85, strong from 68, sound from 48, limited from 30 and weak below that. They are editorial categories. The calculator's half-width, 6 + 14 × (1 − coverage), is a heuristic sensitivity indicator, not a statistical confidence interval or a calibrated measure of extraction error. The ranking debit uses missing methods, editorial context and effect information. While a paper's relevance is unjudged, its rank uses a neutral clinical value (45, or the design ceiling if lower), moved by whichever scales were judged; it shows no clinical score, goes to review, and cannot lead the week. These conventions need external calibration.
Two things are deliberately absent. Citation counts, because at one or two weeks old they measure indexing speed rather than importance. And reader-relevance signals such as Canadian authorship or subspecialty, because they say nothing about how good a paper is; they inform editorial triage instead. Heat is a screening tool, not a verdict. Cold papers can still be great papers.