Article Text

Download PDFPDF

Pharmacoeconomic evaluation of new treatments: efficacy versus effectiveness studies?


The juxtaposition of economic and clinical evaluation raises new issues in the design of clinical trials. Recent pharmacoeconomic guidelines provide some direction, but do not deal with the appropriate timing of economic evaluations in the drug developmental process. Ideally, pharmacoeconomic data should be available at the time of the regulatory and formulary decision making. Current pivotal phase III trials do not provide these data; they are designed to test safety and efficacy (does the drug work under optimal circumstances?) and not to answer questions about the effectiveness of a drug, the more relevant question for economic analysis (does the drug work in usual care?). The use of more “naturalistic” designs for some phase III randomised trials has been suggested. These so called “effectiveness trials” more closely reflect routine clinical practice. They use a more flexible dosage regimen, and a “usual care” instead of a placebo comparator. Patients randomised are more representative of actual practice and outcomes include quality of life and utility measures. They are more suited to provide the data needed to estimate the real benefit of the treatment in actual care. When costs are applied and compared with these benefits, you can estimate the efficiency of allocating resources to this new drug. Increasing the use of effectiveness trials in phase III would decrease the need for economic modelling.

  • pharmacoeconomic evaluation
  • rheumatology

Statistics from

Request Permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

Inclusion of health economic assessment in the development plans for new treatments is becoming increasingly common. This juxtaposition of economic and clinical evaluation raises new issues in the design, analysis and interpretation of phase II and III trials. These issues are particularly relevant to the pharmaceutical industry, regulators, and payers because they challenge the traditional data requirements for drug and formulary approvals; they are equally relevant to clinicians that must make therapeutic decisions based on these data. As rheumatologist, we are currently faced with an explosion of new, but more expensive, drugs for rheumatoid arthritis. How do you trade off differences in efficacy, toxicity and costs of these new drugs when compared with our usual treatments? What data are needed, how and when should these data be obtained in the drug development process?

In the context of limited financial resources, pharmacoeconomic data have become increasingly important for reimbursement decisions. Indeed some countries such as Canada and Australia now require the submission of economic analyses for a new pharmaceutical agent to be reimbursed under their drug benefit programme.1 ,2 The use of economic evaluation in the health sector is comparatively new and its methodology has evolved rapidly. Although several areas still remain controversial, such as the choice of appropriate outcomes,3 ,4 the assessment of future benefits and discounting rates,5-7 and the reporting of uncertainty,8 ,9 there is enough agreement on principles to provide researchers and decision makers with a set of guidelines. These are reflected in the comprehensive recommendations of the Panel on Cost Effectiveness in Health and Medicine10-13and in several guideline issues by specialty journals, public agencies and other experts.14-19

A key remaining issue, not fully resolved by these guidelines, is the appropriate timing of economic evaluations in the drug developmental process. Ideally, from the government and formulary point of view pharmacoeconomic data should be available when a drug is first introduced on the market in order to incorporate this information in the decision making process. Can pivotal phase III randomised trial provide this information? Current phase III trials, designed to test safety and efficacy, may not answer the question about the effectiveness of a new drug. Effectiveness refers to how well a drug performs under real world conditions outside the context of a randomised trial where the experiment no longer holds. Long term observational studies are ideally suited to accurately record the effectiveness of a new drug, its routine care, costs of drug monitoring and adverse event management. Such studies, however, (because of safety and dose monitoring requirements by regulatory agencies) can only be done in post-marketing phase IV studies, too late in the process to assist decisions about reimbursement.

To provide earlier information, several authors20 ,21 have suggested the use of more “naturalistic” designs for some phase III randomised trials where the protocol is relaxed to allow physicians and patients to use the drug and monitor patients in ways that more closely reflect routine clinical practice. These so called “effectiveness trials” differ in many ways from the traditional “efficacy” trial. In this paper we discuss two issues raised by the integration of clinical and economic research in the drug development process, these include the appropriate timing of economic data collection and the major differences between efficacy and effectiveness phase III trials.

Timing of economic data collection

Good economic evaluations require planning, pilot studies and the development of models of disease progression, and drug management. Adequate lead-time and resources are needed to properly execute these studies. Increasingly drug companies start the process early on in their development programme. Phase II studies can be used to pilot data that will subsequently be used in the phase III pivotal studies. Data collection tools for costs, health care utilisation and Quality of Life can be tested, their mean and variance can be assessed in the trial population to assist in the calculation of sample sizes. The frequency of high cost events can be estimated and used to design a more efficient approach to phase III trials where one would focus data collection on high cost events.

Phase III trials are the last chance to collect data before approval and reimbursement decisions. To avoid excessive use of economic modelling it is advisable to include in phase III trials some data collection on Quality of Life, utilities, health care utilisation and costs preferably in the context of some more “naturalistically” designed randomised trial. Health care practitioners, insurers, regulatory bodies and journal editors18 often view economic modelling with suspicion. The early experience with pharmacoeconomics relied on the analysts' speculations combined with expert clinical opinions to estimate what the community drug efficacy, toxicity and health care routine management would be. These, after the fact, retrospective analyses are likely to remain unconvincing, particularly in view of several recently published examples where subsequent real world data contradicted the early assumptions of the economic models.22 More recently we have seen phase III trials that explicitly incorporate economic data collection23; they have the advantage of capturing patient data on outcomes and costs from the same study. These phase III economic data can go a long way in convincing agencies of the true benefit of a product.

Some challenges, however, do remain when trying to tackle economic questions within the design of blinded randomised trials. The competing goals of determining a treatment's efficacy—that is, how it works under ideal circumstances—and treatment effectiveness—that is, how it works in real life—cannot fully be resolved by more naturalistic phase III trials. Despite the increasing numbers of naturalistic phase III trials undertaken by drug companies, phase IV observational studies are still needed. Community care differs in many ways from the care given in centres that participated in the trials. Patients and physicians may be less compliant with the recommendations for drug dose or drug monitoring, patients not included in the randomised trials such as older patients, pregnant women, patients with comorbidities will be exposed to the drug. Unblinded physicians will react differently from blinded physicians to patient's reports of efficacy or adverse events. Issues such as access to care and insurance coverage for some treatments will also influence actual patient management. All of the above will affect estimates of efficacy, toxicity and health care utilisation and the real economic impact of the drug may differ significantly from its premarketing estimate. Indeed some reimbursement authorities are now negotiating agreements, which incorporate data collection from post-marketing studies. These data are subsequently used to reassess the reimbursement policy. Meanwhile, which data can feasibly be collected in phase III trials to shed some light at the time of the initial approval process on the likely economic impact of the drug? What can phase III “effectiveness” trials contribute?

Efficacy versus effectiveness trials

Efficacy trials differ from effectiveness trials in many ways. The contrast between these studies was first pointed out over 30 years ago by Schwartz and Lellouch in a classic paper24 and has been reiterated many times since.25 The introduction of pharmacoeconomic analyses is now reinforcing the need to distinguish between these two approaches. Some of the major differences between these trials are highlighted in table 1. It should be emphasised that the table illustrates two extremes; in reality, most trials will be hybrids with some elements of each; a trial can be classified as one or the other depending on whether it has more elements of one or the other.

Table 1

Efficacy versus effectiveness studies

The fundamental differences between efficacy trials and effectiveness trials relate to objectives and motivation for the trial. The objective of an efficacy trial is to demonstrate that the drug works under optimal circumstances while that of an effectiveness trial is to test how it works under usual practice circumstances. An efficacy trial is undertaken to meet regulatory approval; its design will maximise the potential for detecting efficacy in a population where toxicity is minimised. An effectiveness trial is designed to convince formularies and payers of the actual usefulness of the drug in current practice. To meet these goals several elements of the trials will differ. In one case the drug dose is fixed to maximise efficacy/toxicity ratio, should adverse events occur predetermined titration of the drug are dictated. In the other case, flexible drug regimens are allowed at the discretion of the investigator and in some cases patients may be allowed to switch to an alternate drug. It is then the management protocol that is the subject of the investigation, not the individual treatment. The choice of the comparator is also an important distinguishing feature. In many jurisdictions the regulatory approval encourages the use of placebo controls while for economic appraisal the relevant question is how the new drug compares with the current standard of care. For most countries the benchmark for new treatments in rheumatoid arthritis is currently methotrexate. The Canadian guidelines in addition recommend a comparison against the least cost alternative. To avoid the risk of diluting the effect of the drug, explanatory trials must minimise all potentially confounding extraneous factors; randomisation is essential as well as blinding of both patients and clinician to the treatment. An effectiveness trial may also be blind but it is not always possible, the presence of an adverse event typical of a comparator may unblind the participants. This is not necessarily viewed as detrimental in an explanatory trial, but is accepted as part of physicians' and patients' responses to the treatment and probably reflects the clinical response in practice. Some authors even argue that trials with concurrent economic analyses should be unblinded. They argue that: “ In regular clinical practice, both physicians and their patients know the treatment that they are receiving, which may affect clinical outcomes and costs”. Such trials should for obvious reasons be undertaken only when there is evidence from earlier, blinded efficacy trials that the treatment has potential benefits.

An efficacy trial will recruit as homogenous population as possible to maximise the potential response to treatment; in contrast the design of an effectiveness trial should reflect the range of patients seen in clinical practice. Atypical patients may be included as well as those with comorbidities. This will ensure that patients to whom the treatment will be applied will be represented. Outcome measures may also differ between the two types of trials. In an efficacy trial outcomes directly related to the biological basis of the treatment will be included to confirm that any improvement is mediated by the drugs' mechanism and not by some other external factor. In rheumatoid arthritis this includes the clinical and laboratory measures. In effectiveness trials, outcomes should reflect the range of benefits expected from the treatment that are relevant to the patient and to the payer, these may include improvement in ability to function, quality of life or work productivity. Measuring outcomes in term of clinical end points such as joint swelling or CRP has the disadvantage that the effectiveness of treatments across diseases cannot be compared. The ultimate benefit to the patients is precisely the type of information that the decision maker needs to answer the following question. Is investment in treatment A for disease X a more efficient use of resources that investment in drug B for disease Y? Two types of measures can be used across diseases: quality of life measures and utilities. Most clinicians are now familiar with generic Quality of Life measure such as the SF-36 or the Nottingham Health Profile but are less familiar with utility measures. Utility measures globally rate a given heath status (including all aspect; clinical effectiveness, toxicity, psychological, family, societal and work impact of the disease) on a single metric (that is, 1–100) so health states can be compared across diseases.

Finally, in an efficacy trial patients do not necessarily complete the trial in the group in which they were randomised, but patients are always analysed in the group to which they were initially randomised using an intent to treat approach. This flexibility reflects what actually occurs in practice. In contrast, in an efficacy trial the scientific question of whether the drug under study works or not must be answered and patients should remain in the group in which they were randomised as much as is ethically possible.

Despite these differences in the two approaches, these trials usually reach similar conclusions about the benefit of a treatment. Some aspects of effectiveness trials cannot feasibly be considered at the stage of phase III trials but other may be considered as they more closely resemble the data needed for economic evaluation.

What about efficiency?

This paper has considered the pharmacoeconomic evaluation of treatments so far without any discussion of costs. This was done purposefully to emphasise that the major element of a good economic analysis is a research design that allows the collection of the data needed to estimate the benefit of the treatment in actual care. The measurement of cost is secondary albeit important. Efficiency measures whether health care resources are being used to maximise value for money. Once the potential real benefits of the drug are estimated through effectiveness analyses, then costs can be applied to compare the relation between resources input and ultimate outcomes. Inefficiencies exist when resources could be reallocated in a way to increase the health outcomes achieved.

Summary and conclusions

Good economic analyses of new biological drugs in rheumatoid arthritis will require close collaboration between those involved in study design and implementation and people seeking to use the information generated from the studies. Clinical and economic investigators should be involved early on in the drug development programme, indeed as early as phase II and certainly until phase IV observational studies. Some economic data beyond modelling will be needed by decision makers at the time of reimbursement applications. These data can be obtained through slight modifications of the phase III trials to include elements of effectiveness as well as efficacy in these trials. A full economic analysis at this point in the drug development will still require some element of modelling; standardising the data and methods can most effectively do this. Guidelines are available to assist in the standardisation of modelling approaches; these are essential if we are to compare across several new drugs. The data needed for economic models of rheumatoid arthritis are of such magnitude that combining efforts across analysts can do it most efficiently. Academia can play here an important part by sharing with the industry the extraordinary amount of work already done in the synthesis of the literature on the efficacy and toxicity of competing drugs. Much of this work has been done under the aegis of the Cochrane Collaboration. Actual data on effectiveness will also need to be gathered through large longitudinal observational studies and again this can be done most effectively by combining effort across pharmaceutical sponsors. Implementation of pharmacoeconomic studies will require close collaboration and communication between all parties involved particularly as economic evaluation is a new and evolving field in rheumatology.


Dr A Maetzel is supported by a PhD fellowship (health research) from the Medical Research Council.