In Vitro UV Testing—Robot vs. Human Spreading for Repeatable, Reproducible Results

Editor’s note: This is the third article in a four-part series considering the effects of test variables on SPF results. The final article, to appear in the November 2013 issue, will consider the effects of pressure on SPF results.

Sun protection factor (SPF) is one of the most used indicators for the classification of protection levels afforded by sunscreen products against sunburn, mainly due to harmful UVB radiation (290–320 nm solar spectrum range)—and recent studies,1-3 it should be noted, have shown the importance of protection against UVA (320 nm–400 nm). SPF has historically been evaluated by in vivo methods, but in recent years, for ethical, economical and practical reasons, in vitro evaluations are being used more and more. These methods are recognized worldwide, although with significant disparity in results between, and sometimes even within, laboratories. Further, results from in vitro methods must correlate in vivo, which is challenging mainly due to the biological endpoints of tests being subject-dependant.

One advantage of in vitro methods is that, in theory, they should at the very least provide the same measurement with optimized intra- and inter-laboratory protocols. However, most would agree that this is not currently demonstrated. Indeed, in vitro methods are based on the assessment of UV transmittance through a thin film of sunscreen sample spread onto a roughened substrate, and these measurements are not yet well-controlled.

Recently validated in vitro methods have been proposed and accepted worldwide by the International Organization for Standardization (ISO)4 and Cosmetics Europe, formerly Colipa,5 but they still rely upon an in vivo value.6 Others use a ratio within the absorbance curve, such as the critical wavelength, as proposed by Cosmetics Europe in the European Union. The U.S. Food and Drug Administration (FDA) requires both in vivo and in vitro methods.7

Unfortunately, despite all the hard work, in vitro methods for SPF assessment are still not validated due to the high variability between measurements and disparity within laboratories—even when applying a precise protocol. There is also a major problem in validating any modifications or improvements to in vitro methods, which stir continuous debates. Therefore, the reliability and reproducibility of in vitro results are key to approval and correlation—or at least to sort out reasons for no correlation—and to at last realize a reliable harmonized method. While it seems logical to consider repeatability and reproducibility as a prior condition, many studies are solely focused in vivo/in vitro correlation. Clearly, even in the case of good correlations, it is not enough if similar results between laboratories are not found.

Previous work demonstrates that several parameters must be strictly controlled in order to improve reproducibility; parameters include the amount of product applied,8 spreading protocol,9 topography properties of the substrate,10, 11 and transmittance analyzer used.12 More recent work has identified never before-considered parameters that improve repeatability and reproducibility; i.e., pressure control during the spreading step and substrate surface temperature control throughout the application process.13 Until now, in vitro methods have been based on manual spreading, but it is well-known that mastering all parameters is difficult. Even when applying strict protocol, it is nearly impossible for different laboratories to produce the same results for some products.

Thus, after much development, the authors propose herein a method to significantly improve the repeatability and reproducibility of SPF measurements: automated spreading, realized by means of a robotic arm. This paper investigates the effects of automated spreading on final repeatability and reproducibility of in vitro values. For this, human and automated spreading are compared via a ring test of 36 sunscreen products. Human spreading was led by eight experienced operators from different laboratories, referred to here as Labs A through H; automated spreading was achieved by means of the robotic arm device developed for this purpose.

Since most would agree that strict control over parameters during manual spreading would improve repeatability, and probably also reproducibility, it would have been difficult to compare human spreading within several laboratories under exactly the same conditions. Thus, to provide the best conditions as possible for human spreading, and considering the previous comments for testing, identified parameters were kept strictly identical. The same products were tested in the ring test; the same amounts of products were applied; the same temperature of substrate surface was used; substrates with the same properties—i.e., roughness and surface energy—were used;14 the same spreading methods for movement, pressure and drying time, realized under recordings and video controls, were used; and the same transmittance analyzer was employed.

Furthermore, to avoid any product aging and to be sure that samples were in their best condition, the entire study was completed in less than three weeks, and the workshop was led in the same laboratory, although by individuals from several laboratories. Obviously, as noted, it is well-known that optimal reproducibility cannot be reached when tests are performed in individual laboratories. In fact, international committees have already recognized this phenomenon.

Materials and Methods

Automated spreading: Automated spreading was performed by means of a devicea specifically developed for this purpose by the authors’ laboratory. The first part of the device is a robotic arm that performs precise and repeatable movements, i.e., circular and linear strokes, always with the same pressure. The second part is a tool with a surface hardness that simulates the human finger used to spread the product. A finger cot was used on the tool to prevent contamination and to allow saturation. The combination of these two parts effectively mimics human spreading but with a great improvement in reproducibility.

Previous studies had found the best repeatability and reproducibility with specific polymethyl methacrylate (PMMA) substratesb, thus the device was designed for use with them. The device was fitted with a metallic supportc to control and maintain the substrate surface temperature during the spreading step.10 Images of the device are shown in Figure 1. Finally, in order to reproduce manual spreading conditions, the same application rate was performed for automated spreading, described later.

Substrate selection: The importance of substrate roughness on the reproducibility of in vitro SPF tests has already been demonstrated. Thus, as noted, specific molded PMMA platesb with one face roughened and measuring 47 mm x 47 mm x 1.5 mm were used. Surface topography parameters were controlled by means of a profilometerd, which ensures the plates are in total compliance with the specifications described in the ISO and Cosmetics Europe standards for in vitro UVA-PF determination.4, 5

Sunscreen products: Thirty-six sunscreen products of different types and textures were chosen for this study, including: o/w emulsions, w/o emulsions and oils; and creams, lotions and sprays. These products came from different global companies, and products were not limited to those whose laboratories participated in this study. To estimate proper reproducibility from each operator, a double-blind test was performed for two products tested three times (see Table 1, blue and orange).

Sunscreen application: In order to obtain the correct application rate of 1.3 mg/cm², nine drops equal to 28.7 ± 0.5 mg of the test sunscreen product were applied by a 1-mL syringe across the entire PMMA plate. Immediately after weighing, the sunscreen product was spread over the whole surface using a defined sequence of circular and linear strokes. During spreading, the pressure was controlled by means of a balance, and the spreading movement and time were monitored and assured by means of a video system. Concerning manual spreading, before applying the test products, each researcher’s application finger, without finger cot, was pre-saturated with the product. After product spreading, the sample was allowed to dry and settle for 15 min. During the entire application, spreading and drying process, the temperature of the substrate surface and product/substrate interface was maintained at 25°C by means of the specified temperature devicec. After drying, UV transmittance measurements were performed on the samples.

Transmittance measurements: Evaluation of product absorbance was performed by means of a spectrophotometere. Before measurements, the transmittance analyzer was validated and controlled according to in vitro UVA-PF procedure by both specified standardsf and the linearity/additivity by a calibrated reference standardg to which UV filters were applied. Measurements were taken from 290–400 nm at each 1-nm increment. A PMMA plate covered with a film of white petrolatum, 15 mg applied, was used to obtain blank transmittance.

In vitro SPF calculation: Combining the UV light attenuation by the sunscreen with the erythema action spectrum and a relevant solar emission spectrum, the in vitro SPF can be calculated according to Eq. 1:


where E(λ) is erythema action spectrum (CIE-1987) at wavelength λ, which represents the relative effectiveness of UVR in producing erythema in human skin; I(λ) is the spectral irradiance received from the UV source at wavelength λ in midday, midsummer sunlight; A(λ) is the monochromatic absorbance of the sunscreen layer at wavelength λ, with A(λ) = -log [T(λ)]; and d(λ) is the wavelength step, equal to 1 nm in this study. Finally, the in vitro SPF reported in this study is the arithmetical mean of three plates.

Results and Discussion

The in vitro SPFs of the 36 products evaluated by manual and automated spreading are shown in Table 1 according to mean and standard error mean (SEM) for the eight laboratories and automated spreading. In order to compare the reproducibility between manual and automated spreading, the ring test was reproduced with automated spreading over three different days, referred to here as Auto 1, Auto 2 and Auto 3.

Intra-operator variability: For the first step, intra-operator repeatability via the coefficient of variation (CV) for manual and automated spreading was observed by means of box and whisker plots. Without listing all of the values, this simple and complete univariate representation of quantitative data samples provides a sense of the data’s distribution. For that, several statistical parameters were displayed together, explained in Table 2, according to the CV studied.

Figure 2 and Table 3  clearly demonstrate that automated spreading varied less than the manual spreading. Indeed, a mean 11–16% CV for in vitro SPF was observed between different plates for each operator. Furthermore, a maximum 38–62% CV was observed regardless of the operator. These results confirmed that even if parameters were mastered, several products showed high variability when spread manually. Not surprising, automated spreading allowed for better repeatability between plates, with a mean CV of about 5–8% for in vitro SPF. The maximum CV was about 20%, but only for a few products. Thus, by means of automated spreading, the CV for repeatability was reduced by a factor two.

Concerning the two triplicate blind tests, the in vitro SPFs were slightly more reproducible when applied by the same operator, which shows that even if an operator mastered his/her spreading, while there would be repeatability, there would be differences between several laboratories (see Figures 3 and 4).

In vitro SPF repeatability and reproducibility: The second step compared in vitro SPF repeatability and reproducibility between manual and automated spreading. As previously described, detailed results are shown in Table 1. In order to quantify the level of repeatability and reproducibility, a multiple comparisons test by means of ANOVA Gage Repeatability and Reproducibility (Gage R&R) was utilized. Thus, Figures 5 and 6 show box plots for each operator, manual and automated, and in vitro SPF values for each product. The variation between in vitro SPF values according to products was greater for manual spreading than automated spreading. Indeed, in each case, automated spreading clearly showed higher control over in vitro SPF variability, as indicated by smaller boxes.

Furthermore, statistical analysis of the data can be used to calculate the global percentages of repeatability and reproducibility according to manual vs. automated spreading. Figure 7 shows the improvement of in vitro SPF repeatability (~ 13%) by means of automated spreading, compared with manual spreading (~ 22%). For reproducibility, automated spreading gave a variation of about 10%, compared to about 43% for manual spreading. These results also confirmed that variability is extremely product-dependant, reinforcing the fact that any study realized with too small a quantity of products, as is often the case, will provide contradictory conclusions—especially if based on the correlation of previous product choices. Automation, therefore, appears to be a key factor for both final reproducible results intra laboratory and for identifying bad correlations.

Finally, in order to investigate overall reproducibility, the results from each plate were put through principal component analysis (PCA), which enables researchers to look at the data on a single, two-dimensional map to identify trends or similarities. On this map, the closer the plots are to one another, the more alike they are. In other words, the reproducibility is indicated by means of repartition of plots over map. From Figure 8 the authors demonstrated that in vitro SPF results for manual spreading were widely distributed, indicating poor reproducibility. In contrast, in vitro SPF results for automated spreading were strongly concentrated in one area, which indicates better reproducibility.

As shown in Figures 5, 6, 7 and 8, a clear improvement was observed when using automated spreading; all the products exhibited a drastic reduction of in vitro SPF variation. On the contrary, manual spreading leads to very high in vitro SPF variations between laboratories, as shown by the described international ring tests. After these analyses, the authors could conclude, on a large set of products and wide number of laboratories, an overall improvement in in vitro SPF reproducibility based on the use of automated spreading.


In vitro methods for assessing sunscreen protection could greatly benefit from this automated spreading technique because it addresses one of the most important and overlooked keys to SPF testing: repeatability and reproducibility. Indeed, many methods have focused only on correlations between in vivo and in vitro results without ensuring reproducibility. In some cases, they not only demonstrate poor reproducibility—they go on to perpetually debate the results of the correlations obtained. Clearly, the measurements of these 36 products spread manually by eight different laboratories show a range of variability, which cannot be acceptable for a future harmonized in vitro SPF method. Furthermore, it is important to keep in mind that all these tests were performed in ideal conditions for repeatability and reproducibility; it is well-known that variability will increase when products are tested separately in different laboratories.

After this study, the authors can conclude that automated spreading improves the repeatability and reproducibility of in vitro tests, compared with manual spreading. Furthermore, in vitro SPF values obtained by means of automated spreading were close to the results obtained from manual spreading. The device utilized also allows for automated spreading for compliance with different specified in vitro standards. From here, with automated spreading, it seems that work on improving in vivo/in vitro correlation can proceed without any doubt over the results between different laboratories—bearing in mind that the spreading method and test parameters are product-dependent. In fact, different spreading methods according to product behavior can be defined, and the same results could be expected between laboratories under the same test conditions. This would be unachievable with human spreading, even for a single protocol. Finally, this automated approach provides opportunities to study other, unexpected parameters that could influence in vitro SPF variations by fixing spreading. As an example, the authors have identified the differences in depositing a cream in one vs. several sites. These different aspects will be treated in future works with the help of this new appliance.

Acknowledgements: The authors would like to thank Erik Pourtau and Eric Rosello (A.E.2.I.) for their technical support and all participants of this study: Pascal Bayce and Nadège Silly, of Chanel Parfum Beauté–Pantin; Sylvie Marull and Jamila Essadouni, of Yves Rocher–Issy les Moulineaux; Eric Gooris, Katell Vie and Annick Ferreira of Clarins–Pontoise; Valérie Perier, Hélène Dromigny and Virginie Frances of Pierre Fabre–Castres; José Ginestar, Ingrid Benonge and Florence Doublet of Sisley–Saint Ouen l’Aumône; Belen Gonzalez and Marie Contier of LVMH Recherche Parfums & Cosmétiques–Saint-Jean-de-Braye; and Dominique Moyal, Caroline Tricaud and Modibo Mangassi of L’Oréal–Chevilly-Larue.

1. RM Lavker and K Kaidbey, The spectral dependence for UVA induced cumulative damage in human Skin, J Invest Dermatol 108 17-21 (1997)
2. S Séité, D Moyal, S Richard, J de Rigal, JL Lévêque, C Hourseau and A Fourtanier, Effects of repeated suberythemal doses of UVA in human skin, Eur J Dermatol 7 204-209 (1997)
3. NS Agar, GM Halliday, RSC Barnetson, HN Ananthaswamy, M Wheeler and AM Jones, The basal layer in human squamous tumors harbors more UVA than UVB fingerprint mutations: A role for UVA in human skin carcinogenesis, Proc Natl Acad Sci USA 100 4954-4959 (2004)
4. ISO 24443, Cosmetics–Sun protection test method–Determination of sunscreen UVA photoprotection in vitro, ISO/FDIS 24443: 2011(E) (2011)
5. Cosmetics Europe, Method for in vitro determination of UVA protection, Colipa Guidelines (2011)
6. Cosmetics—Sun protection test method—In vivo determination of the sun protection factor (SPF), ISO 24444 (2010)
7. US FDA, Department of Health and Human Services, Sunscreen drug products for over-the-counter human use, Final Rule 21 CFR parts 201 and 310, Federal Register 76 (117) (Jun 17, 2011)
8. M Pissavini et al, Determination of the in vitro SPF, Cosm & Toil 118(10) 63-72 (Oct 2003)
9. L Fageon, D Moyal, J Coutet and D Candau, Importance of sunscreen products spreading protocol and substrate roughness for in vitro sun protection factor assessment, Intl J Cosmet Sci 1-13 (2009)
10. L Ferrero, M Pissavini, A Dehais, S Marguerie and L Zastrow, Importance of substrate roughness for in vitro sun protection assessment, IFSCC 9(2) 1-13 (Apr/Jun 2006)
11. M Pissavini, S Marguerie, A Dehais, L Ferrero and L Zastrow, Characterizing roughness: A new substrate to measure SPF, Cosm & Toil 124(9) 56-64 (Sep 2009)
12. N Cariou and D Lutz, Sunscreen in-vitro SPF determination inter and intra comparison tests between several measurement instruments. Compendium on sun care, HPC Today 7(3) (Jul/Sep 2012)
13. S Miksa, D Lutz and C Guy, UV transmission assessment: Influence of temperature on substrate surface, Cosm & Toil 128(7) 484-494 (Jul 2013)
14. S Miksa, D Lutz and C Guy, Adjusting substrate/product interfacial properties to improve in vivo/in vitro SPF Correlation, Cosm & Toil 128(3) 170-181 (Mar 2013)

More in Sun Protection