A pilot study is synonymous with a feasibility study intended to guide the planning of a large-scale investigation.
As suggested by an African proverb from the Ashanti people in Ghana “You never test the depth of a river with both feet“, the main goal of pilot studies is to assess feasibility so as to avoid potentially disastrous consequences of embarking on a large study – which could potentially“drown” the whole research effort.
Pilot studies can be conducted in both quantitative and qualitative studies.
In general, the rationale for a pilot study can be grouped under several broad classifications: process, resources, management, and scientific.
Pilot studies can be very informative, not only to the researchers conducting them but also to others doing similar work. However, many of them never get published, often because of the way the results are presented. Quite often the emphasis is wrongly placed on statistical significance, not on feasibility – which is the main focus of the pilot study.
In many cases, pilot studies are conducted to generate data for sample size calculations. This seems especially sensible in situations where there are no data from previous studies to inform this process. However, it can be dangerous to use pilot studies to estimate treatment effects, as such estimates may be unrealistic/biased because of the limited sample sizes. Therefore if not used cautiously, results of pilot studies can potentially mislead sample size or power calculations – particularly if the pilot study was done to see if there is likely to be a treatment effect in the main study.
A study should only be conducted if the results will be informative. The focus of a pilot study should be on assessment of feasibility, unless it was powered appropriately to assess statistical significance.
In general, sample size calculations may not be required for some pilot studies. It is important that the sample for a pilot be representative of the target study population. It should also be based on the same inclusion/exclusion criteria as the main study. As a rule of thumb, a pilot study should be large enough to provide useful information about the aspects that are being assessed for feasibility.
The sample used in the pilot may be included in the main study, but caution is needed to ensure the key features of the main study are preserved in the pilot (e.g. blinding in randomized controlled trials).
In general, the outcome of a pilot study can be one of the following: (i) Stop – main study not feasible; (ii) Continue, but modify protocol – feasible with modifications; (iii) Continue without modifications, but monitor closely – feasible with close monitoring and (iv) Continue without modifications – feasible as is.
Q: Why is it important to publish the results of pilot studies? A: To provide information about feasibility to the research community to save resources being unnecessarily spent on studies that may not be feasible. Further, having such information can help researchers to avoid duplication of efforts in assessing feasibility. Our focus should be on feasibility goals. Emphasis should not be placed on statistical significance when pilot studies are not powered to detect minimal clinically important differences. Such studies typically do not show statistically significant results – remember that underpowered studies (with no statistically significant results) are inconclusive, not negative since “no evidence of effect” is not “evidence of no effect”.
Q: Can the results of the pilot study be valid on their own, without existence of the main study? A: Yes, if the results show that it is not feasible to proceed to the main study or there is insufficient funding.
It is always best for a pilot to maintain the same design as the main study.
Conducting a pilot prior to the main study can enhance the likelihood of success of the main study and potentially help to avoid doomed main studies. They should be scrutinized the same way as full scale studies, and every attempt should be taken to publish the results in peer-reviewed journals.
Source:
[1] Thabane L, Ma J, Chu R, Cheng J, Ismaila A, Rios LP, et al. A tutorial on pilot studies: the what, why and how. BMC Medical Research Methodology. 2010;10(1):1. doi: 10.1186/1471-2288-10-1.