- Review Article
- Abstract
- Background
- Need for Pragmatic Trials
- Pragmatic Trials (PTs)
- Example of Explanatory and the Need for Pragmatic Trials
- Designing Pragmatism: When and How
- Importance of Context and Applicability in Pragmatic Trials
- Implications for Evidence-Based Medicine
- Limitations of Pragmatic Trials
- Conclusion
- Declaration
- References
Pragmatic Trials: Importance and Impact
Amit Kumar1* and Vandna Rana2
1Astra Zeneca Pharma India Ltd, India 2Indegene Lifesystems Pvt Ltd, India
Submission: July 04, 2017; Published: August 03, 2017
*Corresponding author: Amit Kumar, Manager- Trials & Publications, Astra Zeneca Pharma India Ltd, India, Email: amit_bpharma_84@yahoo.com
How to cite this article: Amit K, Vandna R.Pragmatic Trials: Importance and Impact.J Endocrinol Thyroid Res. 2017; 2(4):555594. DOI: 10.19480/JETR.2017.02.555594.
- Review Article
- Abstract
- Background
- Need for Pragmatic Trials
- Pragmatic Trials (PTs)
- Example of Explanatory and the Need for Pragmatic Trials
- Designing Pragmatism: When and How
- Importance of Context and Applicability in Pragmatic Trials
- Implications for Evidence-Based Medicine
- Limitations of Pragmatic Trials
- Conclusion
- Declaration
- References
Abstract
Clinical trials have been the mainstay of research and are considered as the gold standard to test and evaluate interventions over decades. The traditional clinical trials face a major challenge i.e., a gap between the research and the delivery of care. Now days there is a growing need for the development of a high-quality, widely applicable and a more evidence-based approach. Trials are categorized into two types: explanatory and pragmatic. Explanatory trials purpose to test whether an intervention works under ideal situations where as pragmatic trials are designed to evaluate the effectiveness of the trials in everyday practice. The pitfalls of the explanatory trials have found for a more practical approach i.e. pragmatic in attitude. Pragmatic trials provide results that can be more generalized and applied to a broader population and can be used in day-to-day practice whereas explanatory trials cannot be generalized broadly. This review discusses the concept of a pragmatic trial, its importance and impact of this design on health care systems, patients, and providers. Further, this review discusses the differences between the explanatory and pragmatic trials, examples of explanatory trials that need a more pragmatic approach, continuum between pragmatic and explanatory trials along with the limitations of pragmatic trials.
- Review Article
- Abstract
- Background
- Need for Pragmatic Trials
- Pragmatic Trials (PTs)
- Example of Explanatory and the Need for Pragmatic Trials
- Designing Pragmatism: When and How
- Importance of Context and Applicability in Pragmatic Trials
- Implications for Evidence-Based Medicine
- Limitations of Pragmatic Trials
- Conclusion
- Declaration
- References
Background
Over the past few decades, there is a massive and continuous research, evolution, and discoveries going on in the field of health sciences which includes therapies, surgical procedures/delivery methods, diagnostic and prognostic tests [1]. The research involves experiments/trials in the healthcare interventions which provide information on potential public health impacts and are also critical to guide evidence-informed public health decision-making and practice. These interventional experiments/trials help to control and minimize the bias associated with it [2]. Traditional randomized controlled trials (RCTs) were introduced which became the ‘gold standard’ of study designs in providing the decisions about the effect of different interventions, minimize systematic errors by randomization, blinding, allocation concealment, etc. These RCTs became the mainstay of clinical research producing high internal validity and reducing the bias, particularly the selection bias which includes extended inclusion and exclusion criteria [3-5]. Traditional RCTs focus more on the efficacy evaluation of drug therapies but rarely produce findings that are easily put into practice. Participants who participate in the study would benefit from the investigation as the study was conducted in a defined population under ideal conditions. Also these traditional RCTs are slow and expensive which produce findings that cannot be practiced easily. After ~17 years of clinical research, only 14% of the findings have led to the widespread changes in the delivery of care [6]. The study design of RCTs produces credible outcomes when conducted properly, but these results cannot be applied to the real-world settings due to its high internal validity (with proper eligibility criteria, blinding, and controlled environment which minimizes the risk of bias) that hinders the external validity (generalizability and applicability of the trial’s results) and therefore cannot be generalized to other populations and different other clinical settings [7].
- Review Article
- Abstract
- Background
- Need for Pragmatic Trials
- Pragmatic Trials (PTs)
- Example of Explanatory and the Need for Pragmatic Trials
- Designing Pragmatism: When and How
- Importance of Context and Applicability in Pragmatic Trials
- Implications for Evidence-Based Medicine
- Limitations of Pragmatic Trials
- Conclusion
- Declaration
- References
Need for Pragmatic Trials
Table 1.
- Review Article
- Abstract
- Background
- Need for Pragmatic Trials
- Pragmatic Trials (PTs)
- Example of Explanatory and the Need for Pragmatic Trials
- Designing Pragmatism: When and How
- Importance of Context and Applicability in Pragmatic Trials
- Implications for Evidence-Based Medicine
- Limitations of Pragmatic Trials
- Conclusion
- Declaration
- References
Pragmatic Trials (PTs)
The terms ‘explanatory’ and ‘pragmatic’ terms were coined by two French statisticians, Schwartz and Lellouch in 1967 and they were aware of the limited applicability of the RCTs 40 years ago, [8] and proposed a differentiation between the trials. Explanatory/efficacy trials describe the relationship between an intervention and its physiological outcome and evaluate the efficacy of a trial in well-defined and controlled settings, whereas pragmatic/effectiveness trials evaluate an intervention in a broader routine clinical practice. The results of the RCTsare required to be tested under real world settings to assess the impact of an intervention [4]. PTs are considered as a way of bridging the gap between research and care [9]. These ensure the generalizability of a trial and represent the population for which the treatment will be effective [1]. PTs are designed with input from health systems and produce evidence that can be readily disseminated and used to improve care. PTs include health systems, providers, and patients as partners, accelerating the integration of research, policy and practice [2]. Also, these provide wide spectrum of outcomes which are mostly patientcentered.
The pragmatic research involves pragmatic clinical trials (PTs) which are designed to improve practice and policy. Unlike RCTs, PTs are conducted in trial settings where everyday care happens, such as community health care clinics, hospitals, and health systems. These clinical trials include patients from different populations. Idealized situations are not created in PTs. Instead, they work along with patients, physicians, and leaders in the health care systems to develop the study. These trials often start with a pilot phase, and continuing the collaboration till the completion of the trial. Compared to efficacy and explanatory trials, pragmatic trials aim for the results that apply to the realworld care. It is likely that the pragmatic trials show improved results of health and care as they adapt the intervention to the local context and test in typical care settings [1,2].
To increase the value of a trial, decision makers in health system and investigators should follow these approaches to make PTs different from RCTs. Core characteristics of PTs [10-12] include, the intervention should compare with usual care rather than placebo, enroll patients from heterogeneous population to increase generalizability, recruit patients from different communities which means those who are not involved in the research to improve external validity and ensure their representativeness, measure broad range of outcomes to estimate all anticipated effects of an intervention with long-term follow-up, generate evidence that suggests the intervention of the study to be adapted to local context (those interventions designed for individual practices and their markets). Certain aspects of trial design in PTs are different from RCTs. The inclusion and exclusion criteria for PTs are not very restrictive as they are performed in routine clinical environment. Placebos are not given as a part of routine practice, so PTs do not include placebo in their trial design. In PTs, investigators compare different treatment approaches available against a new treatment. It is difficult to perform blinding in PTs. Contrary to the RCTs; biasness of physicians can be accepted sometimes in PTs, because in real world settings patients and therapists’ expectations may influence the size of the treatment effect. In PTs, the patients who do not follow the recommended treatment are also included in the analysis because non-compliant patients are also part of study population. Hence, if an investigator wants to prove that an intervention is effective in real world settings, we suggest more trials with pragmatic design should be conducted. Investigator should provide extra care and attention to the issues of generalizability due to high external validity [1,2].
The advantages of adopting pragmatic approach is that the trials conducted are practical in nature, which are designed to test what works in everyday care. PTs involve all the participants (physicians, patients, health systems) in the trial to draw meaningful conclusions. In addition, PTs have less participant exclusion as compared to the traditional RCTs, which in turn leads to the assessment for larger subset of participants. PTs evaluate measures designed to increase effectiveness. Traditional RCTs follow very tight sample control, which further makes it challenging to generalize the results and vice versa for the pragmatic trials [1,2].
Partnerships in pragmatic trials help in a faster and a more relevant research. These partnerships with healthcare systems help in altering the research and eventually improving the health of the individuals. Also, speed up the scientific discoveries from lab to patients, improve everyday care in community clinics, hospitals, and health systems. Collaborating with providers and organizations as integral partners helps to gain evidence practically on improving patient health and satisfaction [2].
A traditional trial has poor applicability, or at times it is difficult for others to make decisions about its applicability, an opportunity that is lost in influencing the clinical practice and healthcare delivery. Well-designed trials with a pragmatic attitude should be increased in number to help the health systems and providers in designing a trial that helps in routine clinical practice. In addition, it also helps in developing research questions and goals, which are focused more on patient-centered research and care. Hence, this approach will aid in transparent reporting of results that are dedicated on the issues and data that are relevant for making decisions and taking action [2,13]. Table 1 presents the key differences between trials with explanatory and pragmatic attitudes [3].
- Review Article
- Abstract
- Background
- Need for Pragmatic Trials
- Pragmatic Trials (PTs)
- Example of Explanatory and the Need for Pragmatic Trials
- Designing Pragmatism: When and How
- Importance of Context and Applicability in Pragmatic Trials
- Implications for Evidence-Based Medicine
- Limitations of Pragmatic Trials
- Conclusion
- Declaration
- References
Example of Explanatory and the Need for Pragmatic Trials
clinician considering an objective to evaluate treatment for secondary prevention of stroke from the Heart Outcomes Prevention Evaluation (HOPE) trial and may exclude few criteria where patients suffer due to non-adherence, side-effects or consent withdrawal in the preclinical trial phases, and uses placebo as comparator rather than aspirin [14]. The study still requires more trials with wide applicability both from those who are interested in improved treatment for clinical problems [15- 17] and those interested in health policy [10,18].
- Review Article
- Abstract
- Background
- Need for Pragmatic Trials
- Pragmatic Trials (PTs)
- Example of Explanatory and the Need for Pragmatic Trials
- Designing Pragmatism: When and How
- Importance of Context and Applicability in Pragmatic Trials
- Implications for Evidence-Based Medicine
- Limitations of Pragmatic Trials
- Conclusion
- Declaration
- References
Designing Pragmatism: When and How
The first article with the concept of pragmatism was published in 1967. There is an increase in the usage of the terms like pragmatic, practical and naturalistic which require more evidence that it is applicable in routine clinical settings. There is a growing trend of pragmatism now days. Most of the trials include both explanatory and pragmatic aspects [3].
There are two tools available for the trialist to know whether his or her trial have the factual design and evaluate where the trial is best placed on explanatory-pragmatic continuum even though they have some differences in the aims. The first, developed by Gartlehner et al. [21] characterized trials either as efficacy/explanatory or effectiveness/pragmatic, and was designed to organize these trials for systematic reviews and to help clinicians judge the applicability of the trial results. This tool has seven domains to explain the nature, whether explanatory or pragmatic trial. These include the trial settings, its criteria of inclusion, health outcome options and the followup duration, etc. But this characterization failed due to some drawbacks as the tool instead of showing a continuum presented that the trial can either be explanatory or pragmatic. This tool also had an added problem regarding the trial setting in primary care wherein when a trial conducted in a referral hospital cannot be oriented towards asking questions related to the effectiveness of the real world.
Thorpe KE et al. [10], few years later, had introduced the concept or tool of Pragmatic- Explanatory Continuum Indicator Summary (PRECIS) which enabled the investigators to assess the degree to which their designs aligned with the trial’s objective. According to the trial design decision, this tool had 10 dimensions.
- Eligibility criteria
- Flexibility of the experimental intervention
- Practitioner expertise (experimental)
- Flexibility of the comparison intervention
- Practitioner expertise (comparison)
- Follow-up intensity
- Outcomes
- Participant compliance
- Practitioner adherence
- Primary outcomes.
The described 10 dimensions help in quickly finding out the inconsistencies as to how they are managed in a trial. The trialists with the help of these domains make adjustments, accordingly and appropriately to the purpose of the trial to obtain consistency. Koppenaal T et al. [22] in 2011, has reviewed two systematic reviews using PRECIS score adaptation. One trial adapted more pragmatic characteristics while other included more explanatory aspects. Koppenaal T et al. [22] has made few other modifications in the trial. For example, this tool has been adapted in the assessment of systematic reviews, which introduced a scale from 1 to 5 for the 10 domains (1 represents explanatory and 5 represents pragmatic end). Using this scale, they demonstrated that their modification of the tool could help in quantifying the continuum per domain. They also observed that the systematic review that used more pragmatic characteristics had a higher average score in the PRECIS 10 domains [22]. The study by Tosh G et al. [23] in the issue of Dialogues in Clinical Neuroscience adapts PRECIS tool to help the appraisal of RCTs by mental health researchers. They used a scoring system from 0 to 5 (0 -domain cannot be evaluated) to quantify the explanatory continuum and provide recommendations for change of the trial’s score [23].
- Review Article
- Abstract
- Background
- Need for Pragmatic Trials
- Pragmatic Trials (PTs)
- Example of Explanatory and the Need for Pragmatic Trials
- Designing Pragmatism: When and How
- Importance of Context and Applicability in Pragmatic Trials
- Implications for Evidence-Based Medicine
- Limitations of Pragmatic Trials
- Conclusion
- Declaration
- References
Importance of Context and Applicability in Pragmatic Trials
Karanicolas PJ et al. [22] observed that the PRECIS tool does not address the point of how and in what way a trial is pragmatic depends on the context and perspective/perception [24]. These two are important aspects for interpreting a trial result. For interpreting a trial result, both perspective and context are required. Perspective is defined as an individual reading a trial report and is difficult for a trialist to predict the individual’s perception. While context is a feature of trial which may include the trial settings, patients/participants, clinicians, etc and trialists should have the capability to describe regarding these. The trialists should not consider the various perspectives of those making decisions instead consider whether all these perspectives describe the trial context [24-27].
For example, in a Dutch pragmatic trial conducted by Riper H et al. [28] in 2007 assessing web-based self-help for problem drinking, one of the inclusion criteria included participants should have internet access [28]. In 2007, there is an 88% internet penetration in Netherlands [29] while it is 30% in Poland [30]. According to the perception of a clinician or policymaker, the trial conducted in Dutch was more explanatory when compared to the study conducted in Poland due to the limited internet penetration. Another context-specific example of a pragmatic trial conducted in Quebec, Canada which compared the delivery of care by general practitioners with the pharmacistmanaged anticoagulation service [31]. The trialists’ found that the delivery of care was similar in both the groups but found that the pharmacist-managed anticoagulation service was expensive. The authors of the study reported that this is a context specific care by the physicians where they provide the care through telephone consultation and receive no compensation for the physicians in Quebec. Without this contextual based information readers might conclude that this is not applicable to their contexts and may feel the need to improve the delivery of care.
Decisions about applicability solely depend on the readers as to how they take the intervention in their own context [3]. The important contextual information should be properly delivered to the readers without which it is extremely difficult for them to make informed judgments about applicability.
According to the Consolidated Standards of Reporting Trials (CONSORT) Statement extension which was proposed recently, the reporting of pragmatic trials reporting should improve the contextual information reporting [3]. This especially requires the information about the participants and on applicability of the trial findings. Some groups like Workgroup for Intervention Development and Evaluation Research (WIDER) [32], the CONSORT extension for non-pharmacological treatments [33] and the Standards for Quality Improvement Reporting Excellence (SQUIRE) Statement [34] help others to judge the applicability of an intervention to their own setting and can apply according to their selection. A poorly described trial cannot effectively render information and becomes useless. It is also essential that the readers need to know the results and findings on ‘who, what, when and where [35].
Statistical modelling helps in the estimation of the applicability of the trial results. Methods for making judgements regarding generalizability by comparing the participants in the RCTs and individuals in large surveys, epidemiological studies etc has been introduced [36]. The demographic profiles of youths included in trials were found to be mostly similar with survey populations. But in the trial participants, the rate of suicidal ideation and behaviors among the depressed adolescents in the USA was about half the adjusted rate which was assessed from the national database (3.6% vs 7.1%) [36]. This difference was due to the trials excluding adolescents who are considered to be at high risk of suicide. In 1999, Yamaguchi and Ohashi proposed a proportional hazard model for a multicenter superficial bladder cancer trial to assess the treatment-by-centre and baseline risk effects on the results of the trial [37]. They also found that, although there was some differences between the centres, especially in the baseline risk, which showed a slight change in the estimate of treatment effect [37]. The predictive power of modelling should not be overstated on the benefits of an intervention which is applied outside the trial’s original context.
- Review Article
- Abstract
- Background
- Need for Pragmatic Trials
- Pragmatic Trials (PTs)
- Example of Explanatory and the Need for Pragmatic Trials
- Designing Pragmatism: When and How
- Importance of Context and Applicability in Pragmatic Trials
- Implications for Evidence-Based Medicine
- Limitations of Pragmatic Trials
- Conclusion
- Declaration
- References
Implications for Evidence-Based Medicine
The idea of generalizability and applicability in PTs benefit the health sciences community. PTs include the sensitization of policy makers, practitioners and even patients and further involving them in the research culture till the end of the trial and this becomes point becomes the most advantageous aspect of PTs. Few trials require more pragmatic aspects and few others require more explanatory aspects. The post-hoc exploratory analyses of the PT trial results still needs to be evaluated by explanatory trials and many other trials include in continuum between the PTs as well as explanatory.
The two-important evidence-based medicines include the RCTs and the systematic reviews [38]. The important tools for synthesizing data include systematic reviews and metaanalyses which provide extensive data heterogeneity. Systematic reviews and meta-analyses with PRECIS score help in systematic mapping of pragmatism in published research [1]. Evidence from meta-analysis of multiple treatments with the suitable statistical techniques also helps in evaluating the effectiveness [39]. Medical journals can adopt pragmatic aspects of the trials with the help of CONSORT extension or PRECIS [9].
- Review Article
- Abstract
- Background
- Need for Pragmatic Trials
- Pragmatic Trials (PTs)
- Example of Explanatory and the Need for Pragmatic Trials
- Designing Pragmatism: When and How
- Importance of Context and Applicability in Pragmatic Trials
- Implications for Evidence-Based Medicine
- Limitations of Pragmatic Trials
- Conclusion
- Declaration
- References
Limitations of Pragmatic Trials
Need for adjustment between internal and external validity
The main limitation of PTs include, to gain the external validity for a more real-world applicability of the results, it may reduce the internal validity. The results provided by PTs are more generalizable to other settings and contexts, and the intervention should be delivered as planned and if not so or not uniform across practices, it may be difficult to detect the effects of intervention [1].
For example, an explanatory trial, which is robustly designed and performed with a well-studied intervention, which can be effective in a particular combination of practitioners/patients but can be less effective in extended populations. This, when executed under broader real-life settings causes a dilution of effect. RCTs conducted on moderate treatments might have advantages by blinding, concealment, allocation, patient preferences, etc. which effects the outcomes of the treatments. But when these studies conducted in a pragmatically yield a more statistically significant result when compared to RCTs [1].
Requirement of adequate resources
Significant costs are associated in conducting the PTs. These include
- Implementation of random assignment in real-world settings with varied populations involving practices and patients,
- Collecting comprehensive outcomes data, and
- Measuring implementation carefully.
Documenting implementation for such large data is critical in determining the actually tested and to explore the outcomes across practices [1].
The cost of PTs can be enormous due to the implementation of large sample sizes and long follow-up periods to produce reliable and practical evidence [1]. For instance, the ALLHAT (Antihypertensive and Lipid-Lowering Treatment to Prevent Heart Attack) trial [40], a well-planned RCT evaluating 4 antihypertensive treatments took 8 years to complete the trial and the cost was more than $100 million [41].
Intensifying difficulty of specifying in the delivery of exact intervention
A third limitation is that a PT may make it difficult to find out the effects of different components associated with the intervention in different settings. A PT approach allows for more heterogeneous interventional components where practices are adjusted according to the context of specific clinical settings. Some components show pragmatic interventional effects whereas others do not show any effect and some may vary in practice settings [1].
A RCT of endarterectomy for recent carotid stenosis from the European Carotid Surgery Trial (ECST) [42] was exemplified by Rothwell [43]. The clinical settings were different between the countries due to which a raised heterogeneity was observed in the new stroke investigation time which in turn affects the overall effectiveness of the endarterectomy. Also, there is an indefinite evidence as to whether the clinical settings within the same countries were comparable and show no sign if the treatment is effective in one clinical setting provide similar effectiveness in another and vice versa.
- Review Article
- Abstract
- Background
- Need for Pragmatic Trials
- Pragmatic Trials (PTs)
- Example of Explanatory and the Need for Pragmatic Trials
- Designing Pragmatism: When and How
- Importance of Context and Applicability in Pragmatic Trials
- Implications for Evidence-Based Medicine
- Limitations of Pragmatic Trials
- Conclusion
- Declaration
- References
Conclusion
A trial with internal validity and poor applicability is of no use in clinical practice and is difficult for others to make judgments out of these trial results. Hence, an emerging and upcoming concept of pragmatic trials was developed where trials are conducted in real-life settings that encompass the full spectrum of population to which an intervention is applied. The results of pragmatic trial help in conducting the trials in large populations with high external validity. The scientific community, physicians, policy makers, and health care recipients, involved in the PTs should be aware of the idea of pragmatism and mandate results with a more evidence-based approach in real-life settings. These pragmatic trials help in understanding the differences associated with the trial protocols that contribute to the heterogeneity and may contribute the clinicians with the ability to determine how best to apply a new therapy. But few trials require a continuum of both the concepts (explanatory and pragmatic trials) to answer the complicated problems lying ahead of us. In conclusion, the results of a pragmatic trial is directly applicable to clinicians, patients, payers, and health systems which can help decision makers to improve policies and practices with answers to all the questions related to the risks-benefits and intervention costs.
- Review Article
- Abstract
- Background
- Need for Pragmatic Trials
- Pragmatic Trials (PTs)
- Example of Explanatory and the Need for Pragmatic Trials
- Designing Pragmatism: When and How
- Importance of Context and Applicability in Pragmatic Trials
- Implications for Evidence-Based Medicine
- Limitations of Pragmatic Trials
- Conclusion
- Declaration
- References
Declaration
Amit Kumar and Vandna Rana are employee of Astra Zeneca Pharma India and Indegene Lifesystems Pvt Ltd respectively, however, both have written this paper based on their personal capacity.
- Review Article
- Abstract
- Background
- Need for Pragmatic Trials
- Pragmatic Trials (PTs)
- Example of Explanatory and the Need for Pragmatic Trials
- Designing Pragmatism: When and How
- Importance of Context and Applicability in Pragmatic Trials
- Implications for Evidence-Based Medicine
- Limitations of Pragmatic Trials
- Conclusion
- Declaration
- References
References
- Patsopoulas NA (2011) A pragmatic view on pragmatic trials. Dialogues Clin Neurosci 13(2): 217-224.
- Treweek S, Zwarenstein M (2009) Making trials matter: pragmatic and explanatory trials and the problem of applicability. Trials 10: 37.
- Zwarenstein M, Treweek S, Gagnier JJ, Altman DG, Tunis S, et al. (2008) Improving the reporting of pragmatic trials: an extension of the CONSORT statement. BMJ 337: a2390.
- Yoong SL, Wolfenden L, Clinton-McHarg T, Waters E, Pettman TL, et al. (2014) Cochrane Update Exploring the pragmatic and explanatory study design on outcomes of systematic reviews of public health interventions: a case study on obesity prevention trials. Journal of Public Health 36(1): 170-176.
- Flay BR, Biglan A, Boruch RF, Castro FG, Gottfredson D, et al. (2005) Standards of evidence: criteria for efficacy, effectiveness and dissemination. Prev Sci 6(3): 151-175.
- Balas EA, Boren SA (2000) Managing clinical knowledge for health care improvement. Yearb Med Inform 1: 65-70.
- Nallamothu BK, Hayward RA, Bates ER (2008) Beyond the randomized clinical trial, The role of effectiveness studies in evaluating cardiovascular therapies. Circulation 118(12): 1294-1303.
- Schwartz D, Lellouch J (1967) Explanatory and pragmatic attitudes in therapeutical trials. J Chronic Dis 20(8): 637-648.
- Califf RM, Sugarman J (2015) Exploring the ethical and regulatory issues in pragmatic clinical trials. Clin trials 12(5): 436-441.
- Thorpe KE, Zwarenstein M, Oxman AD, Treweek S, Furberg CD, et al. (2009) A pragmatic-explanatory continuum indicator summary (PRECIS): a tool to help trial designers. J Clin Epidemiol 62(5): 464- 475.
- Tunis SR, Stryer DB, Clancy CM et al. (2003) Practical clinical trials: increasing the value of clinical research for decision making in clinical and health policy. JAMA 290(12): 1624-1632.
- Glasgow RE, Magid DJ, Beck A, Ritzwoller D, Estabrooks PA (2005) Practical clinical trials for translating research to practice: design and measurement recommendations. Med Care 43(6): 551-557.
- Eldridge S (2010) Pragmatic trials in primary health care: what, when, and how? Fam Pract 27(6): 591-592.
- Bosch J, Yusuf S, Pogue J, Sleight P, Lonn E, et al. (2002) HOPE Investigators: Heart outcomes prevention evaluation. Use of ramipril in preventing stroke: double blind randomised trial. BMJ 324(7339): 699-702.
- Rothwell PM (2006) Factors that can affect the external validity of randomised controlled trials. PLoS Clin Trials 1(1): e9-e10.
- Hotopf M, Churchill R, Glyn L (1999) Pragmatic randomised controlled trials in psychiatry. Br J Psychiatry 175: 217-223.
- Marson A, Kadir Z, Chadwick D (1997) Large pragmatic randomised studies of new antiepileptic drugs are needed. BMJ 314(7096): 1764.
- Lavis JN, Posada FB, Haines A, Osei E (2004) Use of research to inform public policymaking. Lancet 364(9445): 1615-1621.
- Bombardier C, Laine L, Reicin A, Shapiro D, Burgos-Vargas R, et al. (2000) Comparison of upper gastrointestinal toxicity of rofecoxib and naproxen in patients with rheumatoid arthritis. VIGOR Study Group. N Engl J Med 343(21): 1520-1528.
- Bresalier RS, Sandler RS, Quan H, Bolognese JA, Oxenius B, et al. (2005) Cardiovascular events associated with rofecoxib in a colorectal adenoma chemoprevention trial. N Engl J Med 352(11): 1092-1102.
- Gartlehner G, Hansen RA, Nissman D, Lohr KN, Carey TS (2006) A simple and valid tool distinguished efficacy from effectiveness studies. J Clin Epidemiol 59(10): 1040-1048.
- Koppenaal T, Linmans J, Knottnerus JA, Spigt M (2011) Pragmatic vs. explanatory: an adaptation of the PRECIS tool helps to judge the applicability of systematic reviews for daily practice. J Clin Epidemiol 64(10): 1095-1101.
- Tosh G, Soares-Weiser K, Adams C (2011) Pragmatic vs. explanatory trials: the PRAGMASCOPE tool to help measuring differences in protocols of mental health randomized controlled trials. Dialogues Clin Neurosci 13(2): 209-215.
- Karanicolas PJ, Montori VM, Devereaux PJ, Schünermann H, Guyatt GH (2009) A new “mechanistic-practical” framework for designing and interpreting randomized trials. J Clin Epidemiol 62(5): 479-484.
- Oxman AD, Lombard C, Treweek S, Gagnier JJ, Maclure M, et al. (2009) Why we will remain pragmatists: four problems with the impractical mechanistic framework and a better solution. J Clin Epidemiol 62(5): 485-488.
- Karanicolas PJ, Montori VM, Devereaux PJ, Schünermann H, Guyatt GH (2009) The practicalists’ response. J Clin Epidemiol 62(5): 489-494.
- Oxman AD, Lombard C, Treweek S, Gagnier JJ, Maclure M, et al. (2009) A pragmatic resolution. J Clin Epidemiol 62(5): 495-498.
- Riper H, Kramer J, Smit F, Conijn B, Schippers G, et al. (2007) Webbased self-help for problem drinkers: a pragmatic randomized trial. Addiction 103(2): 218-227.
- Internet World Stats Netherlands Internet Usage Stats and Telecom Reports.
- Internet World Stats Poland Internet Usage Stats and Telecom Reports.
- Lalonde L, Martineau J, Blais N, Montigny M, Ginsberg J, et al. (2008) Is long-term pharmacist-managed anticoagulation service efficient? A pragmatic randomized controlled trial. Am Heart J 156(1): 148-154.
- Intervention Design.
- Boutron I, Moher D, Altman DG, Schulz KF, Ravaud P, et al. (2008) Extending the CONSORT statement to randomized trials of nonpharmacologic treatment: explanation and elaboration. Ann Intern Med 148(4): 295-309.
- Ogrinc G, Mooney SE, Estrada C, Foster T, Goldmann D, et al. (2008) The SQUIRE (Standards for QUality Improvement Reporting Excellence) guidelines for quality improvement reporting: explanation and elaboration. Qual Saf Health Care 17 (Suppl 1): i13-i32.
- Glasziou P, Meats E, Heneghan C, Shepperd S (2008) What is missing from descriptions of treatment in trials and reviews? BMJ 336(7659): 1472-1474.
- Greenhouse JB, Kaizar EE, Kelleher K, Seltman H, Gardner W (2008) Generalizing from clinical trial data: a case study. The risk of suicidality among pediatric antidepressant users. Stat Med 27(11): 1801-1813.
- Yamaguchi T, Ohashi Y (1999) Investigating centre effects in a multicentre clinical trial of superficial bladder cancer. Stat Med 18(15): 1961-1971.
- Patsopoulos NA, Analatos AA, Ioannidis JP (2005) Relative citation impact of various study designs in the health sciences. JAMA 293(19): 2362-2366.
- Cipriani A, Furukawa TA, Salanti G, Geddes JR, Higgins JP, et al. (2009) Comparative efficacy and acceptability of 12 new-generation antidepressants: a multiple-treatments meta-analysis. Lancet 373(9665): 746-758.
- ALLHAT Officers and Coordinators for the ALLHAT Collaborative Research Group, The Antihypertensive and Lipid-Lowering Treatment to Prevent Heart Attack Trial (2002) Major outcomes in high-risk hypertensive patients randomized to angiotensin-converting enzyme inhibitor or calcium channel blocker vs. diuretic: The Antihypertensive and Lipid-Lowering Treatment to Prevent Heart Attack Trial (ALLHAT). JAMA 288(23): 2981-2997.
- Goldberg NH, Schneeweiss S, Kowal MK, Gagne JJ (2011) Availability of comparative efficacy data at the time of drug approval in the United States. JAMA 305(17): 1786-1789.
- Masuhr F, Busch M, Einhaupl KM (1998) Differences in medical and surgical therapy for stroke prevention between leading experts in North America and Western Europe. Stroke 29(2): 339-345.
- Rothwell PM (2005) External validity of randomised controlled trials: to whom do the results of this trial apply? Lancet 365(9453): 82-93.