Repeat yourself

Pharmaceutical & Medical Packaging News staff

Revised ISO standard stresses test method validation.

By John Conroy

Talk about your stress tests. Changes to a key international standard have put pressure on medical device manufacturers to verify the methods they use for validating their packaging tests. The revised standard, ISO 11607 1 2006 for terminally sterilized medical devices, has been in effect for roughly one year. Despite efforts made to clarify package-testing and test-method-validation duties, some confusion exists over how to effectively use the test methods and achieve repeatability and reproducibility.

“The revision of ISO 11607 has put more emphasis on the notion of validated test methods,” says Patrick Nolan, COO of DDL Inc. (Eden Prairie, MN), a package testing service. “The device manufacturers have had to spend more time and effort in their own testing labs to validate the test methods they perform.”

“Validation is definitely critical,” says Nick Fotis, director of the packaging technology center for Cardinal Health (Dublin, OH), who also serves as an editorial advisory board member for PMP News. “There’s a misperception in the industry that if you’re using an ASTM method that has gone [through] the trouble of validating [and] is a true round-robin [procedure], [you can] just pick up this test method and [be] validated. That’s not true at all.”

In the year or so since the new ISO version was introduced, many of the speakers at packaging conferences have focused on test method validation, says Hal Miller, president of the consulting firm Pace Solutions (Cape May, NJ). In the past, factors such as the pass/fail aspect of packaging-integrity methods and the destructive nature of seal strength tests have made validation difficult, he notes. In particular, destructive tests do not have “a base reference to go by. As a result, test method validation in our industry prior to a couple of years ago, I would say, has been very lax.”

Miller, chairman of ASTM Committee F02 on Flexible Barrier Packaging, traces the evolution of current validation methods to a concerted effort by leaders in the device industry to convince FDA “that physical test methods were more sensitive than microbial tests. Primarily, we wanted to get away from sterility testing for packaging integrity. As a result, we convinced FDA that physical tests were as sensitive as or more sensitive than microbial whole challenge tests.” FDA subsequently wrote a memorandum “that in essence said physical tests can be used as an alternative method to do microbial challenges for packaging integrity,” Miller says.

FDA placed five conditions on that approval—number one being “that test methods must be validated,” Miller notes. One of the goals of the ISO standard is to establish the need for validations. As Fotis points out, Miller says that manufacturers can’t just pick an ASTM study off the rack. “Just because someone like ASTM…has already done an interlaboratory study [that] has reproducible and repeatable data doesn’t mean that it’s validated, because you as the user must validate it in your own laboratory,” Miller emphasizes.

“The ASTM method will give you a start,” Miller says. “For example, what type of sensitivity do you expect? If it’s a 50-micron hole, then you should produce a 50-micron hole to see if you can duplicate it. It also gives you the reliability or repeatability, and therefore that should be your criteria for success: Do as well or better than that.”

Nolan notes that the revised benchmark “has provided a list of test methods in Annex B that can be used for providing compliance with the standard. However, not all the listed standards have precision and bias (P&B) statements based on round-robin testing. Although a P&B is important to provide information about the robustness of the test, this alone does not constitute a validation. The validation must be done by the individual lab to show that the lab can produce data at least as robust as the standard did during the round-robin testing.”

In order for ASTM test methods to have associated repeatability and re­liability—or precision—established, manufacturers need to conduct a round-robin study, also called an interlaboratory study (ILS), Miller says. ASTM E691 provides guidance on how to run an ILS that includes a recommended number of participating laboratories, different samples, and sample replicates.

An example of a round-robin study might entail taking a NIST-certified reference film and having a number of technicians analyze the same film across many modules, says Joel Fischer, lab manager for Mocon (Minneapolis), an instrumentation and lab services supplier. For example, Technician A runs tests on instruments 1 and 2, Technician B tests the film on instruments 3 and 4, Technician C tests the film on modules 5 and 6, and so on. The “big question,” he says, is, “What is the repeatability of my lab?” This type of study answers that question. Fischer says his lab “primarily does permeation testing” and the industry commonly conducts round-robin verification of ASTM standards by sending “common samples to different labs” around the country for comparison.

In a presentation at Healthpack 2007, Randall Troutman, senior engineering manager of Oliver Medical (Grand Rapids, MI), pointed out that all test methods that are used to show compliance with both parts one and two of ISO 11607 “shall be validated and documented.” The standard’s Annex B contains a list of suitable test methods. Both parts require the validation to demonstrate the suitability of the method used. The validation shall also establish both a rationale for the selection of the appropriate packaging system test and for acceptance criteria, according to Troutman, who said the revision is a “hot-button issue” that’s caused some confusion in the industry.

Repeatability and reproducibility are, of course, the twin pillars of test method validation. Repeatability refers to the variability of independent test results from within a single laboratory, while reproducibility addresses variability from results among different labs.

Annex B contains a list of standard guides, practices, and test methods from several worldwide organizations. In addition to ASTM and ISO, the organizations are EN, USP, TAPPI, and DIN, according to Troutman. Each organization offers a range of test methods. ASTM’s tests include peel, dye, and bubble emission, while ISO’s include air permeability and biological evaluation, and USP covers biological reactivity and in vitro methods.

Jon Anderson, director of quality for Alcan Medical Flexibles Packaging (Chicago), says that repeatability and reproducibility analysis is one of the best methods for determining test measurement error. The easiest way to interpret the results could be to think of them in terms of humans and machines, he says. Humans can reproduce, and therefore are the cause of “reproducibility” errors, while machines make the mistakes in “repeatability” errors, he points out. Differences in testing methods (or the execution of that test method) are one possible reason for human errors, Anderson says. In that case, training, visual aids, and other means can be used to improve the test methods by working “on the human aspect,” as he puts it.

Machine errors may have environmental or related causes that can be improved by examining the precision and accuracy of the test machine, Anderson says. For example, tensile strength [testing] equipment with a 1000-lb load cell won’t have the precision “to measure tensile seal strength of a peelable seal, say in the 1 to 5 lb/in. range.” If that’s the case, then it’s time to change or improve your test equipment, he says.

The other factor involved with repeatability and reproducibility studies is whether or not you are performing destructive testing. If you are unable to test the same sample twice, you must attempt to reduce the sample variance on the results when obtaining samples, Anderson says. For instance, thickness tests designed to allow two operators to test the same spot without destroying the package can lead to more repeatable results.

“These tactical situations give you a percentage error rate,” Anderson says. “Once you know your error rate, you’ll know how valid your results are.”

The lack of standard values in testing causes consternation for manufacturers, says Stephen Franks, executive vice president of T.M. Electronics (Boylston, MA), which sells combined seal-strength and leak-integrity testers. The first question packaging customers ask is, “‘What’s the standard value I should be looking for?’” he says. “For either a leak test or a burst test, there are none in this industry. It’s totally dependent on the nature of the product or package. You have to first get that message across, which is not easily accepted.

“Everyone would like to have a standard to work to, of course, because it would eliminate uncertainty in validation issues,” Franks continues. “However, because the variations inherent in products and packages make a universal standard impossible to define, the medical packaging industry is forced to rely on using ISO 11607 to support product specifications such as sterility and tensile strength.” Three times a week, customers will call asking for advice on how to proceed, says Franks, who notes that the issue “is really one of educating our customers.”

“We are familiar with the ISO guidelines, and we talk with customers about qualifying our testers for their process,” Franks says. “We offer IQ/OQs (installation qualification/operational qualification procedures) to help the customer,” he notes, pointing out that the PQ (performance qualification) involves assigning an engineer to work with the customer on validation matters. Taking this step can add significant costs for the customer over and above the actual cost of the package testers themselves, however.

“I tell clients of mine that if you can use a test method that complies with an existing standard already, especially an ASTM standard, half of your battle is almost done,” says Mike Troedel, president of the consulting firm Troedel & Associates (Lake Villa, IL). “Generally, most of the tests in our industry are standard tests. Unless you’re trying to do something internally that’s really unique, you can usually find a consensus method that you can reference.”

Troedel says more than 95% of the industry takes that approach. He cites an example of a manufacturer needing “to measure the opening strength of a tear feature” on a device package. In this instance, the usual seal strength test would be ineffective. “You’d have to write your own in-house method, have to validate that method, validate its repeatability and reproducibility, and you’d have to have all the data to support that.”

In his Healthpack presentation, Troutman of Oliver Medical pointed out that nonstandardized test methods are acceptable. However, using them has at least three drawbacks, he said: They provide no data for validation, do not ensure that the test is measuring the intended attribute, and do not satisfy U.S. regulators’ expectations for data from the validation process.

The increased time that device manufacturers have spent shoring up their labs’ quality and technical expertise has come at “considerable expense,” DDL’s Nolan says. “The commercial testing labs have experienced the same level of burden and have been scrutinized more aggressively by medical device manufacturers. Those labs that do not pass manufacturer audits are not added to Approved Vendor lists.”

Troedel says all the “big guys— Cardinal, Johnson & Johnson, Boston Scientific—were always” validating tests. He’s not as certain about the compliance rate of the smaller companies—an opinion shared by other experts.

Manufacturers that neglect to follow ISO 11607’s recommendations do so at the risk of FDA sanction, insists Fotis of Cardinal Health. “FDA has recognized ISO 11607 as a consensus standard,” he says. “Nevertheless, it’s voluntary. If I as a packaging engineer get audited by FDA, I can show the auditor that FDA recognizes my approach is legitimate as long as I am in full compliance with ISO 11607.”   

“It’s not just a regulatory thing,” he concludes, “but it’s a wise thing. If there were ever a lawsuit against a device manufacturer, a plaintiff’s attorney could very well bring up the fact that there’s an international standard—ISO 11607.”

And that’s a stress test no one wants to face.

John Conroy is a freelance writer based in Los Angeles.