2018 Sentinel Initiative Annual Public Workshop


American Today News

(people chattering) – Good morning, everyone. I’m Mark McClellan, I’m the director for the Duke-Margolis Center for Health Policy, and I’d like to welcome all of you today in person and on our webcast to this annual Sentinel Initiative Public Workshop, which is being hosted by the Duke-Margolis Center for Health Policy and supported by a cooperative agreement with the FDA. This is a special meeting. This year marks the 10th anniversary of the Sentinel Initiative Annual Public Workshop. Along with that, the 10th anniversary of work at FDA in collaboration with a broad range of private sector organizations, entities, researchers, industry, and others to implement an post-market surveillance system at the FDA.

Some of you here today have attended previous workshops, which have provided a venue for communicating what’s going on in Sentinel and what the opportunities for further developments are. And also, ideas for bringing new approaches into Sentinel. Some of you go back further than that. Looking around the room, I see many people who were involved in initially standing up this collaborative system for developing real-world evidence related to safety and an increasing array of other types of evidence on medical products.

Some of you were here at the beginning, back when the legislation enabling the Sentinel System was passed with strong bipartisan support, led by Senator Kennedy and Senator Enzi back in 2007. And some of you were there even before that. Some of the ideas, before there were terms like real-world evidence made their way into the potential for a national system like this. And I’m sure there are many people here who are going to see this program into a future that has an even bigger impact through developing real-world data and real-world evidence to impact the effective use of medical products. So there has been a lot accomplished, but we know that there is a lot of potential to further advance Sentinel into a robust 21st century system that generates timely and useful real-world evidence on a range of evidentiary needs. There also are some challenges to address to achieve this aim, and with the development of new data sources, new ideas, more experience, more methodologic expertise, and more potential for collaboration, continued growing pains, even for an approach that is now a decade old. Addressing these challenges is going to require continued efforts across stakeholders, like what were are trying to do today with the active participation of payers, health systems, industry, academic partners, and so on, to refine the Sentinel Initiative’s capabilities.

In fact, today, you even got a glossary of terms, along with your meeting materials, recognizing just how broad Sentinel has become, with terms like AV and BEST. For those of you who aren’t familiar with those, you will be by the end of the day, as you hear more about how the Sentinel program is extending. And this includes considering the emergence of innovative tools like blockchain, cloud computing, artificial intelligence tools that hold considerable potential for expanding the impact of this program, but again, have not yet been fully developed and applied. Today we want to discuss and explore how innovative ideas like these can benefit from Sentinel’s existing capabilities, which you’ll hear about and hear illustrated, in terms of the many recent achievements of the program.

And we’re also gonna talk about a long-term path towards modernizing and improving the suite of tools that not only leverages new technologies, but continues to build on the underlying partnerships and collaborations that have made Sentinel’s model possible, and that have made it so successful. We’re looking forward to a robust discussion on all of these issues today, as we focus on Sentinel’s continued development during this meeting on the Sentinel Initiative. Today, we’re very honored to host Commissioner Scott Gottlieb, who will give a keynote address to get us going. First, I wanna quickly go through the agenda for today and a few housekeeping items. After the commissioner’s keynote address, the Sentinel leadership from across FDA centers, the Drug Center, the Biologic Center, the Device Center, will highlight achievements and discuss their respective centers’ priorities for the coming year.

We’ll then move into a session that will feature a behind-the-scenes look at the Sentinel System operations, from key figures who are guiding its use, including the principal investigator of the Coordinating Center, Rich Platt, and the leadership from a few of Sentinel’s diverse data partners. After that, we’ll take a short break. And then coming back, we’ll hear from CDER and CBER Sentinel leads on some key achievements and plans for the further development of Sentinel’s tools. We’ll then have panelist reactions from representatives of the FDA’s Office of New Drugs and private industry on how they’re leveraging Sentinel’s tools and results in their work. There’s a break for lunch.

After lunch, we’ll reconvene to discuss ongoing efforts expanding access to and use of the Sentinel System in increasingly diverse ways, beyond the direct use by FDA for some of these important safety issues and effectiveness issues, now, too. We’ll then take a quick break before our last session of the day, which will be a full panel with what I think will be a very lively discussion on what could be in store for Sentinel over the next 10 years. I know you’re all, so before we get to the commissioner’s address, just a few more housekeeping items. I want to remind everyone that this is a public meeting that is also being webcast online. We welcome all of our webcast participants. The record of this webcast will be available on the Duke-Margolis website following the event. I also want to emphasize that this is not only an opportunity for everyone to hear about what’s going on from FDA and other stakeholders involved in the Sentinel program, but an opportunity for thoughtful questions and comments from all of you as well. We’ve set aside time in each session to address comments and questions throughout the day. As a heads-up to our speakers and panelists, Isha Sharma and Nick Fiore, who are right here in front, Nick, Nick will be right there.

Nick and Isha will be here with cards to indicate how much time you have left on your presentation. We have a lot to cover, so we are going to stick with the timelines. For those of you who are joining us, please feel free to help yourself to coffee and beverages outside the room throughout the day. A reminder that lunch will be on your own, there are a number of restaurants in the area, and we can give you some tips on those, too.

And we’ll be starting, again, these sessions on time. Feel free to bring food back into the room here. Okay, enough on housekeeping, now to get to the substance. I am very pleased to hand things over to Dr. Scott Gottlieb, the Commissioner of Food and Drugs at the FDA. Scott has brought a unique combination of skills and experiences, some might say he was made for this job, in terms of the public policy engagement, the expertise on FDA-related issues, the perspectives from private sector experience, as well. He has been, and I hope he talks about this a little bit, a strong leader and advocate, not only for improving public health through evidence and through making the most of FDA’s capabilities, but also through collaboration with the private sector.

And as I think you’ll hear, has been involved in the early stages of the development of this initiative, back when it was a concept and back through the legislative process that got Sentinel enacted. He was Deputy Commissioner at that time, and he’s also taken many steps to expand the use of real-world data and real-world evidence at FDA. I’ve known him for a long time and I can’t think of a better person to address this event at such a critical time, 10 year anniversary, of the Sentinel Initiative. Scott, please come on up. (audience clapping) – Thanks a lot, thanks for having me today. When I first started working at FDA in 2003, as a senior advisor to Dr. McClellan, I worked on drafting a lot of his public remarks. And there was one speech in particular, where I wrote about how we needed an active surveillance system. We needed to move away from MedWatch. And I remember all the consternation I got internally, because there was a perception that we were very far away from being able to achieve that.

We’re still not quite there yet, but it’s hard to imagine, just the short period of time, how much closer we’ve gotten to that vision of having a truly active surveillance system, using big data and the technologies that have become available since that time. I think in 2003, that was probably a vision that didn’t comport with the state of the technology at that time. But I think a lot of people at FDA in the Office of Drug Safety, in CDER, Janet Woodcock, others, saw the vision, invested in it and made it come about. It’s a real pleasure to be with you today, to reflect on the past and speak about the future of what is one of the most important initiatives at FDA in collaborations ever to be undertaken by the agency.

On a personal level, I’ll tell you it’s especially meaningful for me to join you on this 10th anniversary workshop, because Sentinel played such a significant role during my initial tenure at FDA. Although Mark and I weren’t at the agency when Sentinel came into full fruition, we were involved, as he said, in its early years, and in some of the work leading up the launch. This is especially gratifying to see how the promise of this project has been realized. And today plays a very important role inside the agency. With that in mind, I want to offer just a few opening thoughts about how Sentinel has developed into such a successful data infrastructure.

And that today, provides a truly 21st-century surveillance system for the agency, and how we might build on that achievement going forward. Sentinel is a powerful tool for FDA by creating the scientific capacity to recognize and address safety issues in a timely fashion, by giving FDA a broad and deep access to electronic healthcare data that extends all the way back to the medical record and the patient encounter. And by standing this up in cost-effective and scalable way, in a scalable system, FDA’s been able to enhance public health safety in previously unimaginable ways. Sentinel has allowed us to develop innovative ways to monitor FDA regulated products and answer numerous questions simultaneously. And do it in a matter of weeks and months rather than years that it might have taken in the past.

The idea for Sentinel was based on a very simple principle, one that goes to the heart of FDA’s science-based mission. We realize that to keep up with the accelerating pace of scientific discovery, we needed to develop high-level tools that would allow us to apply and translate the data and use those products in a real world information that would benefit patients and consumers. We realize the limitations from a passive system, the passive way that we gathered a lot of our data and safety issues. And Sentinel was chance to be much more active in how we collected safety information by tapping into real-world observational data that could be queried to answer specific scientific questions.

Sentinel has also been a critical piece of our comprehensive response to patients and caregivers who want and deserve more and timelier information about the benefits and potentials risks of the products they use. Sentinel is one of several successful initiatives that FDA developed to drive innovation and incorporate emerging technologies into everything we do to protect and promote the public health. By linking projects like Sentinel and the National Evaluation System for health Technology, or NEST, which CDRH is working on with other database initiatives at our sister agencies, and in organizations outside the government, we’ve been able to build a rigorous and robust foundation for producing more useful knowledge at a fraction of the cost required from previous efforts. Today, Sentinel has become one of the largest and most successful uses of big data in healthcare, a much more efficient system that can generate much more high-quality evidence in a shorter period of time.

And though it continues to develop, we already have realized many of the benefits that come from the ability to use real-world observational data a tool to monitor medical product safety and identify further concerns and be able to evaluate them. Those benefits can extend in the future as well. For example, we should consider how Sentinel might be used to answer questions about efficacy, and how FDA might have the tools and resources to take on these questions in certain narrow circumstances, where a question around a product’s efficacy also relates to its safety. One such situation is when it comes to the long-term efficacy of opioid drugs and the long-term prescribing of these medications. For example, if the chronic use of an opioid drug is marked by diminishing efficacy from these medicines, then long-term prescribing might lead to higher and higher doses being prescribed over time. That situation could in turn lead to more dependency, and in some cases, more addiction.

This clinical question is something that we ought to be able to fully understand. Right now, we don’t have good data on the long-term efficacy of opioid drugs, even though they’re prescribed for these purposes. By leveraging large databases containing electronic health records and disease specific registries and claims data, we’ve made significant advances in our understanding of health and disease and safety issues, and the relationship between health-related factors and outcomes. You’re going to hear today plenty of statistics on Sentinel, but let me offer a few of the impressive numbers that frame some of this success. There were more than 223 million members in 2016, and 178 million members with information from both the medical and pharmacy benefits.

There are 43 million people currently accruing new data. There are 425 million person years of observation time. And there are billion unique encounters, all contained within this system. And FDA has done 254 analyses using the Sentinel Common Data Model and reusable modular program since full activation of the Sentinel System in 2016, including 116 in 2016 and 138 in 2017. None of this would have been possible without the enormous contributions of our data partners who’ve helped us build Sentinel. Their sharing of data and scientific expertise allowed FDA to fulfill its mission, its important public health mission. By joining together, they’ve enabled the creation of this invaluable resource for public health.

And their continue collaboration has allowed us to expand and improve the capabilities of Sentinel and transform it into a true national resource. The Sentinel System is a model for how FDA can work with diverse elements of the healthcare industry, researchers, patient groups, and all to try to accelerate the pace and the impact of post-market safety activities. This type of real-world evidence infrastructure is providing useful data for other activities and other studies by other organizations including IMEDS and the studies of effectiveness, the FLUVAC study, that could lead to both new knowledge and changes in effectiveness-related thinking and labeling about products. We’re committed to continuing this model of collaboration and we need to advance this infrastructure. It’s become critical to producing the high-quality evidence needed to guide FDA’s decisions about products it’s charged with regulating, and help us better inform patients. We know that to meet our growing evidentiary scientific needs through access to these data resources, this tool must also continue to evolve.

And one of the key questions you’ll be addressing today is just how does FDA and its collaborators build on the success of Sentinel to date? The increasing innovation of IT systems and cloud-based technologies, the use of blockchain and artificial intelligence, and other types of advanced computing offers continuing opportunities to advance Sentinel’s development and improve the system. And improve its data and analytic capabilities, advances for post-market safety surveillance as well as the growing range of research needs. Our PDUFA VI commitments for Sentinel charge us with increasing the sophistication of the core capabilities, especially in areas of pregnancy safety, signal detection, analytic tools, and filling gaps in data sources to improve the sufficiency of the system to address safety questions. We’re also committed to improving the system’s transparency in how we communicate with industry and the public with what we find. Sentinel System is leading the way to faster, better, cheaper analyses proving how real-world data is a valuable resource for the whole of healthcare enterprise. And so this workshop provides an opportunity to discuss the possibilities and solicit ideas from across the stakeholder community on how to achieve such aims over the next decade. And I know that several of my colleagues will be addressing these points in more detail as this day goes on.

So I hope your discussion of these topics is productive today, and I hope we can benefit from your help. I want to again thank all the partners for their commitment to this effort, each of you is really key to the success of what we’re doing here. And each of you plays a vital role in finding new solutions to protecting and promoting public health. Thanks a lot. (audience clapping) – All right, thank you, Commissioner, it’s great to hear that perspective. Both on the past and the future. And as you heard, Scott’s been involved in this set of activities for a very long time. I’d like to ask our panelists for the next session on the Sentinel Initiative, what does the next 10 years hold, to come up now while I introduce them. Glad to see everybody made it in, despite the weather this morning. Let me introduce, first, Gerald Dal Pan, the Director of the Office of Surveillance and Epidemiology at the Center for Drug Evaluation and Research at FDA.

Steven Anderson, the Director of the Office of Biostatistics and Epidemiology at the Center for Biologics Evaluation and Research at FDA. And Greg Pappas, Associate Director for National Device Evaluation at the Center for Devices and Radiological Health at FDA. We are gonna hear from these three leaders from across the FDA centers on how the Sentinel System is affecting the regulatory missions and the work that they are able to do. And also with a look ahead. So how they’re benefiting from the use of Sentinel, some illustrative milestones of what it’s achieved and contributing to the mission of each of the centers, and their perspectives partner what the path forward looks like for Sentinel. So you see this recurrent theme during the day of understanding where we’ve been, where we are, with an eye towards where the Sentinel Initiative broadly can go.

And as you’ll hear, there’s an increasingly diverse range of activities underway related to the Sentinel Initiative. So Gerald, I’d like to start with you, please. – So good morning, Mark, and thank you. Thank you, everyone, for coming today. What we heard both from Mark and Scott a little bit about the history going back about 15 years now, I think when we recognized the need to move our post-market drug safety system beyond our MedWatch program, beyond spontaneous reports. Those reports are still actually quite important for what we do and account for probably over half of our post-market safety label changes. But nonetheless, they’re limited in the scope of the kinds of questions they answer. To answer some of the more complex and subtle questions, we needed a better system. One that was able to look at large populations of people, put identified adverse medical outcomes into the context of how a drug is used, and be able to do so in a timely and efficient way.

So while we’ve been using administrative claims data for decades, the timely and efficient wasn’t really part of that system. Everything we were doing was reinventing the wheel, so to speak, every time we needed to take a drive. So in 2006, the Institute of Medicine issued its report on the future of drug safety, and had many recommendations, which really have been implemented at FDA. And it called for us to get better data.

And in early 2007, we held the first meeting that I can remember used the word Sentinel, where we just talked in very broad terms about what such a system could be. And we listened to representatives of academia and industry, as to what the potential might be. The passage of the Amendments Act in 2007 gave us the mandate to create this system. And we’ve come a long way since then. The first part of the effort was really organizational. How do we envision important questions such as privacy in informatics contributing to this effort? And what are those considerations? And so now we have the Sentinel System.

We had the Mini-Sentinel Pilot, and then in 2016, so two years ago, we really launched the full-fledged system and have integrated it into our post-market safety operations. We’ve had a lot of training of staff, we have a lot of processes in place, whereby staff are aware of this system, and can use it to answer the questions they had, questions that they might not have been able to answer five or 10 years ago.

And we’ve been able to take advantage of a system that is timely and efficient. The timeline from identifying a safety issue to having a usable, actionable result is much shorter now than it used to be. And in doing so, it leads to cost-effectiveness. This has been enabled by a distributed database model, with periodic updates to the data, as well as to a suite of reusable programs that the safety team at FDA works with their colleagues at the Sentinel Operations Center, and other collaborators to parametrize. When we find that we need an analysis that’s not part of the suite of reusable tools, we can expand that suite of reusable tools, depending on the different needs we have. So Sentinel is not a static system, it’s a dynamic system. It’s one that keeps growing and is modernized to meet our needs. And even with these successes, we still have a long way to go. We’ve not really used Sentinel yet to improve signal detection, we’ve used it more for signal strengthening and signal analysis. But signal detection is something we want to do.

And you’ll hear Josh Gagne later today to talk about that. We recognize also that a lot of what we’ve done so far in Sentinel has been using data from administrative claims, and we need to go beyond that. We need to see how we can harness more of the information available in electronic health records. Information that goes beyond what’s in a typical administrative claim. And we’re working to explore how we can do that now. We’ve also worked hard to make the system publicly available. And I think you’ll hear about IMEDS later today, Claudia Salinas, from Eli Lilly will talk about some industry experience with IMEDS. And then we’re also strengthening international partners, partnerships, our Canadian colleagues have developed CNODES, and you’ll hear what that stands for later. But it’s essentially a Canadian Sentinel, and they’ve adopted the Sentinel Common Data Model. And you’ll hear from Robert Platt later today. So be careful, there are two R. Platts who work in this space.

And then what does the next 10 years hold? Well, in 2019, we’re gonna have to recompete the contract for Sentinel and toward that end, we recently issued a request for information. A request for the public to tell us how we might do things differently, or more efficiently. And specific things we’re interested in are using electronic health records more efficiently, developing technologies such as machine learning and artificial intelligence, natural language processing, and how we can improve access to quick, accurate chart review. And we also have many commitments regarding Sentinel under the Prescription Drug User Fee Act, and we will be using those to fill some gaps with signal detection, pregnancy safety, gaps in data sources, and all the like. So finally, one of the things that the Institute of Medicine said broadly about post-market drug safety, it’s really best when post-market safety and pre-market efficacy safety in review are working together. And Sentinel, so we’ve made lots of efforts at FDA in that regard very broadly. And as we embark on the initiatives of the 21st Century Cures Act passed about 14 months ago, we’ll be seeing how we can leverage the Sentinel infrastructure for issues of real-world evidence.

You’ll hear a little bit today about FDA-Catalyst, and its move in that effort. And so I think in sum, we have here a lot of progress, on a long-term public health and regulatory initiative. And there’s still more room to go. I think that’s what we’re going to be talking about today, so thank you. – Thank you very much, Gerald, Steve? – Sure, So I’m gonna sort of divide my comment into two parts.

I was gonna talk first really about some of the accomplishments that we’ve made with the Sentinel Harvard Pilgrim program. I’ll say that in 2017, it was a productive year for us and that program. We continue to use and integrate Sentinel into our CBER regulatory processes. We’ve dramatically increased the use of the ARIA tools and done a number of queries, I think upwards of 40 more queries, which is a large number for us. And we expect to do more of that in the future. And we completed 10 protocol-based activities, which are sort of the larger epidemiological studies, the safeties of products. Another thing we did that was unique was we launched a rapid surveillance system for the annual influenza vaccine. And it monitors several safety outcomes, along with an effectiveness outcome. Very important, given all that’s going on with influenza this particular year. We’ve hosted a number of trainings, I think we’re up to five trainings this year, for CBER staff to get them more engaged in the Sentinel program, and to integrate that into their thinking about using it for their regulatory needs across the center. Another thing we did was we stood up a governance board.

So we stood up a Sentinel advisory committee, and that has membership from my office, Office of Biostatistics and Epidemiology, and we manage the program within the center. Along with the three product offices, so for vaccines, blood, and then the tissues and advanced therapies. So we’re actively engaging the center staff in as stakeholders in the product offices into the Sentinel program. Also engaging the center director, because the center director is part of the decision process that goes into governance. So we’re very excited about that.

So just to close in this section of comments, we’ve had really, I would say good success with the Sentinel Harvard program. And our lead for Sentinel, Azadeh Shoaibi, will talk a bit more about the accomplishments during her presentation later in the morning. So I wanted to move on to a second set of comments, which really focuses on our continuing efforts to build a better Sentinel System for CBER. The Sentinel Harvard program has been great. It’s addressed some of our regulatory needs, but you know, in the spirit of always wanting to do better, we felt we needed to improve Sentinel and continue to develop an even better surveillance system.

At last year’s annual Sentinel workshop, I emphasized sort of two points that were aspirational in nature. And the first one was sort of the mantra of better, faster, cheaper, which circles again back to this concept that we need to build better and better systems. And then the second point was furthermore aspirational, and that was about automation, specifically automation of adverse event reporting. So I’ll talk in a moment about our efforts in that area. So as far as the progress we’ve made in sort of those two thematic areas from last year, today I’ll announce that we’ve launched the CBER Sentinel BEST system. So we started this program in summer of 2017. BEST stands for the Biologics Effectiveness and Safety system, and the goal really centers around that claims data, but it really is focused around electronic health record data to conduct biological product safety and effectiveness studies. And also automation of adverse event reporting. So after an open and competitive contracting process, we awarded two separate contracts in September 2017 to establish BEST. The contracts were awarded to IQVIA, which is formerly QuintilesIMS. And they partnered with OHDSI, which is the Observational Health Data Sciences and Informatics group.

So this really again, establishes this second program, and in addition to Sentinel Harvard Pilgrim for us. We have the Sentinel BEST program. So playing on words a bit here, we went from wanting a better and better system to wanting the best Sentinel system. So let me just get into what the contracts really deal with. So contract one, I’ll cleverly call it contract one, our goals with contract one are to build a more efficient electronic health record based data system for CBER, so that we can rapidly conduct queries and epidemiological studies for blood products and vaccines. Blood products, this is a very big need for us. And it’s very different from the realm of drugs. So CBER has some fairly different needs, so this is why we’re moving some of these new unique systems and different approaches with our products.

So how is what we’re doing any better and faster? As to getting to better, faster, cheaper. IQVIA, the BEST contractor, provides CBER with an expanded data infrastructure. So I think we’re onboarding about 50 to 70 million patient EHR records. So that’s a large number, considering those types of systems. And then we’re also bringing on board an additional 160 million patients worth of claims data. So again, we’re bringing on these data and these efficiencies. We’re also bringing on the OHDSI and IQVIA data analytic tools that leverage automation. Which fits in with my sort of second theme. So staff can use the OHDSI, for instance, the OHDSI Atlas tool and IQVIA tools from their laptops to design a query from start to finish. That can be a query or an epidemiological study. They can press a button, send that program to IQVIA and IQVIA can run that against the databases we request. So this speeds things up significantly, and it’s transformational, we predict, for the type of work we’re gonna be doing in the future.

It’ll cut things down from even weeks to months to days to weeks. And so we think that’ll bring incredible efficiencies. It’s still in the early phases, but we’re expecting to gain great efficiencies from this. Now I wanted to talk a little bit more about contract two. The primary goal of the second contract is to develop and use innovative methods like data mining, artificial intelligence, machine learning, natural language processing, and others, to identify adverse events associated with exposure to blood products. Within EHR data systems, and then automatically report those adverse events identified into FDA reporting systems like the FAERS system.

The goal eventually is to expand the network. We’ll start with blood products and then expand into vaccines and our other products. So essentially, our goal and aim with contract two is to build the next generation of tools and systems for our future surveillance work for the next sort of five to 10 years of Sentinel. And that’s really our major goal and thrust with this new program BEST. And Alan Williams, Dr. Alan Williams, will be talking a bit more about this in the last session of the day. As far as progress with BEST, I’ll just say in the four months that the program’s been underway, we’ve stood up a query system and run about 100 simple queries. So we’re proud to have that system up and running in such a short period of time, and expect to be able to do even more complex queries in the future.

I’ve talked about better, faster, and cheaper, and then just wanted to touch on the cheaper aspect, and how are we containing costs. So I think I will say that everybody on this panel and within the leadership of FDA are concerned about cost and how do we control costs for these types of systems? Because they’re very expensive, and we all recognize that. So with Sentinel BEST, we’re reducing costs and we’re expecting, at the end of the contract, to achieve a savings of about two to three-fold over sort of previous contracts we have. And so, again, efficiencies and then cost-savings are really critical to us. So in summary, we have added this new CBER Sentinel BEST system. In it’s an EHR-based system, uses new tools and we’ll expect it to expedite the work we do in safety, surveillance, and effectiveness. The other thing is accessibility. So we’re hoping, again, to move these systems out to the public, once we develop them, that they’ll be accessible to the broader community of academics and industry to use.

And then sort of on the final notes, priorities for the next year. We’re continuing to expand and improve Sentinel, and then advance the CBER Sentinel program to begin and continue to generate high-quality data for real-world data, for real-world evidence to support requirements for the User Fee PDUFA act, as well as 21st Century Cures. And then I just wanted to thank the number of people involved in this work. Azadeh Shoaibi and her CBER Sentinel team, the staff at my office, from the business team to the medical officers that do the review, and the center director, who’s been so supportive of the work that we do, Dr. Peter Marks. And then thanks to our Harvard Sentinel colleagues and Rich Platt for the work they do. And then welcoming our new IQVIA and OHDSI colleagues and thank them for the work they do.

And then finally, the thankless job that the data partners have, we thank them for their work, and the academic partners, because they do the yeoman’s work, and it’s a very important part of this whole program. So it can’t be underestimated. And with that, I’ll stop. – All right, Steve, thanks for covering so much ground, next, Greg. – Thanks so, is that on? – It should be. – Thanks so much for this opportunity to give you a view of what things look like from the device world. First of all, it’s been an honor for me to work with Gerald and Bob Ball, and the rest of the Sentinel team over this process. As I’m sure you know, so we’re going from BEST to NEST now.

As I’m sure you know, NEST, the National Evaluation System for health Technology, health with a little H, so it’s NEST, has taken a very different strategy to use real-world evidence for evidence generation in the device space. This is not to be contrary. There are three critical reason I want to emphasize that kind of motivate the rest of my remarks, critical reasons why we have taken this strategy. First of all, is the UDI, second is the specific data needs in the device space, and the third is a strategic effort.

So in the UDI, I’m sure you all know that the drug space, has had a UDI, a unique drug identifier in claims data since the ’70s, we’re only getting to that point right now in devices. We have a law that has the device uniquely identified on the label, but that’s not in claims data, it’s not in most medical records at this point, so we’re still a long ways away to use those sorts of tools. Second is the specific needs in the device space for evaluation. So a lot of devices, when you’re trying to evaluate how they’re, you know, if there’s a problem, it’s not always the device, it’s the person, or the system, or the hospital environment that affects the use. So you’ve got to be able to ferret out the difference between a user effect a device effect. That requires a lot of information that’s typically not available in the systems we’ve been talking about. The third is a strategic issue, and having to do with the fact that, as Mark knows, we don’t have a lot of money. And we’ve taken a very different approach building the system with a vision as we do our work at CDRH, working with regulating devices.

Opportunities come up and we take advantage of those opportunities to build this system. And I’ll tell you a little bit more about that. Despite these constraints, or the needs of devices, we’ve been highly successful in using real-world evidence. We have over 40 regulatory decisions, either hardcore regulatory decisions, label expansions, post-approval studies, 522s, compliance decisions, have all been supported by real-world evidence. We’ve also gained access to over 100 million records in this process, and that encapsulates our two of our strategic goals at CDRH, increasing access to real-world evidence and using real-world evidence. Strategically, we’ve also pursued a number of different approaches simultaneously. We’re looking at the broad long-term issue, but I’m gonna focus more on the short-term issue, the short-term strategies. We’re working with health systems, like the Mercy healthcare system, working with EHRs, they have a highly-evolved, very sophisticated EHR system that becomes very appealing. We’ve also worked with Specialty Societies and their registry. So in the device space, we have the unique confluence of needs. Some of our most, our devices of greatest interest, those that are higher risk, are implantables that surgeons keep registries on.

So we have a resource, and we’ve been, to use the term of art, we’ve been using coordinated registry network, CRNs, which is a linked registry. You start with a registry that provides you with a cohort of people well-exposed, and the device uniquely identified. So what’s an epidemiological study? A cohort, an exposure, and then an outcome. So we’ve linked the registries with outcomes data, and in most cases, it’s been claims data. So we have these CRNs, we’ve got about 16 CRNs now in the making and a very bright future for this part of what NEST is gonna look like. I’m gonna tell you a little bit about one of those case studies, the TVT, the transcatheter valve therapy.

So this an innovation that came up in cardiology. And we worked with the American College of Cardiology and the Society for Thoracic Surgery that maintain this registry to identify this cohort. So it’s almost everyone who has received a valve in this way over the last many years, 100,000 people. And then we linked it to claims data for Medicare. So again, the happy confluence here is that everybody, almost everybody who has this device, is on Medicare. And then we have the Medicare claims, so we’ve obviously validated all those end points and found it very useful. There have been about 20 decisions, regulatory decisions, supported by the TVT registry. And I’ve recently completed with a work group, and I see Ted Heise in the audience and some of my colleagues, Jesse Berlin, a work group with industry and the registries, to evaluate the value created by this approach, this CRN approach.

And the 20 decisions cost industry, there are three companies that make these devices, cost industry about $25 million, so it’s a lot of money. But we costed out what it would have costed, the counter-factual, if we did not have the registry. How much would industry have spent to get these decisions through the FDA, if they did not have the registry? It’s about $150 million, so $100 million savings, 500% return on investment. We also looked at time saved, between months and years saved, so that’s important both for public health, because people have devices that, that get devices that wouldn’t get them otherwise, because of the label expansions, and so it’s faster to market. And also faster to market, which is another benefit to industry. Not to be dismissive or to lapse into irrelevance here, in Sentinel, Sentinel still has a very important role in this system that we’re imagining.

Again, as a linkage, take the case of orthopedic devices. There are registries that give us cohorts of people who have been exposed to orthopedic devices. What we need are then outcomes. Did that person have another device, another joint replacement, five years, 10 years down the road? There is a perfect role for Sentinel to provide it with us that outcomes, this highly-refined, highly-curated system that already exists for us. So we’re kind of working on that. Administratively, say a little bit about NEST, we have set up a coordinating center with, we have, with a contract to MDIC. They have hired an executive director, Rachel Fleurence, who is leading a board that has been recruited. And they are in the process of establishing data partners, designating demonstration projects. They have a couple contracts out looking at the sustainability of the system. This is a kind of, all this in devices has kind of been pay-as-you-go system, so we’re trying to figure out how to keep that system sustainable.

And then also a contract to do evaluation of our system as we look forward to the next round of MDUFA. So with that, I’d like to thank my colleagues here on the table and turn it back to Mark. – Great, thank you all very much for covering so much ground in a limited amount of time. I heard a few key themes highlighting, for one, the unique needs, and data and methods issues at each of the centers related to the products that you’re involved in regulating. This recurrent theme in various forms of better, faster, cheaper, making sure that the Sentinel Initiative overall keeps up with developments in the world of electronic data and the analytics that can be used to interpret it for regulatory purposes. And maybe to create better, faster, cheaper by using Sentinel as an infrastructure for many other real-world evidence applications. And some new directions, like perhaps a greater emphasis coming on signal detection. I heard some discussion about use for effectiveness studies as well. We do have a few minutes for questions from those of you who are here.

There are microphones set up in the audience. Gerald, maybe just quickly to get this going, you mentioned, I think, an use of Sentinel for an approval, could you say a little bit more about that as one of these new directions that Sentinel may be taking? – So I think we really talked more about effectiveness, and we haven’t really broached the issue of approval yet. But my colleague, David Martin, and others at FDA have launched something called FDA-Catalyst.

And they have study called IMPACT-AF. And that study there is looking, I believe, in a randomized way at the impact of providing certain information to people about atrial fibrillation. And so with the randomization there, it’s not about one medicine to another, or a medicine to a placebo, but it’s about randomization to an intervention. So that’s our first step here. And I think this is gonna have to go step by step. Clearly, Cures has asked us to look more at how we can do this, and we’ve got a variety of committees set up, we’ll be having public meetings and other things to really address this. And IMPACT-AF is done using the Sentinel infrastructure. So I think Michael Nguyen will mention that in one of his slides, and David’s here as well, to answer questions about that. – Great, thank you, And Steve, you highlighted in your remarks the importance of medical record data for many of the CBER regulatory issues.

Any early returns on the use of BEST, the augmentation of the Sentinel Harvard approach to incorporate electronic record data? Are you pretty optimistic there, or is gonna take some very different kinds of approaches? – So we’re optimistic, we’re only four months in. So it would be a big task to have much in the way of progress in the medical chart review area. I think what we’re looking at is some nice tools that IQVIA and OHDSI have, which is there is a beta pilot to really automate and semi-automate the chart review process. So we’re really hopeful about that. And it is very necessary for our products, because for vaccines, if there’s a signal, you really wanna be very certain that that signal is a real signal.

And that’s really why we go to the medical chart reviews. And the same for our other products as well. So it’s really a critical part. But I can’t offer any more in the way of progress, at this point, but next year, we’ll have more for you. – Okay, and Greg, for you, you highlighted how different the both context issues and data issues are for devices. But I’m wondering if you looked down the road, beyond the coordinated registry networks that you’re building out today, that have the reliable information on devices and the context in which they’re used, particularly implant and major devices, and the long-term outcomes that come from claims. Over time, it sounds like we’re gonna be getting more electronic medical record data incorporated in the Sentinel System overall. There is a path, I’m not sure exactly where it stands with CMS right now, but there’s a path for more payer support for incorporating UDI in claims, and potentially a number of health systems are starting to incorporate UDI information in their electronic records.

Do you see a convergence here down the road? – Well, first of all, I want to put that question in the context of what we can’t do currently, which is that a lot of devices are never gonna be in registries. As I said, there’s this heavy confluence of high-risk devices that tend to be in registries. There are many important devices that will never be in registries, morcellators, multiple use devices, duodenoscopes, that are all pain points for us. So yes, it’s very important to look at these emerging systems. The sticking point’s still gonna be, you know, there’s EHR, and then there’s digital systems. So EHRs, currently, as we think of them, are never gonna have enough granularity to pick up that information, we’re always gonna have to augment it with something.

But yes, we’re definitely looking at a long-term perspective, a more integrated system. But as I say, since we’re kind of paying for this as we go and we’re, all these decisions that we, all this activity has been, I shouldn’t say this, but kind of on the backs of the regulatory system. It’s paid for, most of it’s been paid for by industry towards a decision. And we’ve been promoting it, because it’s better, faster, cheaper for them, and that’s worked. So that’s kind of what’s driving our train. – And I would encourage anyone who does have a question to head up to the microphone, okay, good, thank you. Can you identify who you are when you ask your question? – Sure, my name is Laurie Griggs, I’m running for the United States Senate in Pennsylvania.

I worked for Reagan at the White House. Three questions, but Mark, you took two of them. – Okay, good, we have only limited time. So good. – Thank you. So nobody’s got a crystal ball, nobody really can understand Donald Trump’s tweets, but, except maybe Ivanka, but what did he mean on Monday night at the State of the Union, when he said we’re gonna get drug prices down, yes, we’re gonna speed time from bench to bedside? I was a nurse when Reagan got shot, by the way, for James Brady.

So what really, really, really, in this extraordinarily contentious political world we live in can we do to really finally, and the EHR, I help hospitals compare Cerner, and Epic, and MEDITECH, and Compucare, and McKesson, and GE, and Allscripts. So possible to spend millions and billions each hospital. – So is the question about how the? – What’s Trump mean when he says you guys are the top executives at the FDA. So you already do know what you’re trying to do over the next four to eight years. What are you really gonna do to get drug prices down? Because hospitals are spending a freaking fortune automating? – Okay, so question about drug prices. And that is an issue that FDA has taken steps to address.

I’m not sure how many of them are part of this particular initiative right now. But better, faster, cheaper, in terms of evidence about to use drugs and other medical products seems like is relevant to this topic? – So yeah, I agree that the better, faster, cheaper, more efficient ways to learn about things more accurately, and in a more timely way, is just generally part of our public health mission. – And any other comments on the same issues? – I would echo the same thing that Gerald says as well. – Yeah, in terms of another issue of where people are concerned about access and cost now, with the really serious toll that the flu has had this year, you mentioned, Steve, using Sentinel in new ways to learn about how safe the vaccine is, how effectively it’s working.

Again, it gets at this better, faster, cheaper issue. Maybe you could say a little bit more about that. – Yeah, so CBER has been, so we monitor flu vaccine on an annual basis. And we’ve done this for the last 10 years, looking at Guillain-Barre syndrome, which was really, if you go all the way back to the swine flu in 1976, that was really a problematic adverse event. So we set up these rapid surveillance systems, so we could rapidly understand if there are adverse events. And then implement any sort of interventions we can. And so with the Sentinel System, we’re trying to set up the same thing, where we can actually just rapidly get that data. And we have four outcomes that we’re looking at, as far as safety, and then I believe we’re looking at hospitalizations as well. – Hospitalizations is more of an effectiveness issue? Preventing a hospitalizations related to flu? – Yeah, so looking at flu cases that go into the hospital as an indication of poor effectiveness of the vaccine.

So again, this in its early years, but we expect to kind of expands these types of programs. Because we feel, obviously, they’re very important for monitoring public health. – And then one more followup on better, faster, cheaper, and using drugs and other medical products more effectively. We’ve talked some about the expanding use of electronic medical record information. Greg, I think you alluded to, and it’s been part of some Sentinel pilot efforts in recent years, use of other types of data. Maybe data coming from new medical devices, maybe data coming directly from patients. Do any of you wanna comment on the potential applications or any activities around these types of new data sources? – I’ll take a first crack.

So some of the end points that are critical for the evaluation of devices, and drugs also, are not very reliable in our current electronic medical systems, let’s look at the case of orthopedic devices. One of the key outcomes is pain and ability to move. And that’s difficult to clinically capture in our current systems. What we’ve been doing is, through mobile apps, and thank you to Duke-Margolis for an outstanding product from this summer. We had a workshop this summer on the mobile app business. Registries have been using mobile apps to connect with patients to collect this kind of more granular, more contextual information about pain. And the lesson that we’re learning is one data set by itself, standalone, is never gonna be probably enough to answer our data needs.

So linkage of multiple modalities, multiple sources, is the way forward, and certainly that’s working for us. I know CDER has been a real pioneer using mobile apps, too. – Right, so we have some products with mobile apps. We have a recent approval of a product to look at adherence to medicines. And I think that when we started talking about this stuff 15 years ago, we probably didn’t envision, you know, your watch having all your heart rate and stuff on it. And so I think that represents the next frontier of this. – Yeah, well, I think a good note to end on, again, thank you all for covering a lot of ground in what has happened and what’s coming in the Sentinel Initiative. Really appreciate all of you joining us, thank you. – Thanks. (audience clapping) – All right, now I’m gonna stay up here, just move over to the podium for a second to introduce the next panel. And I’d like to ask all of them to come on up.

So our next session is on updates on Sentinel’s network of collaborators, focusing on this core program, as you heard about with the Sentinel Harvard initiative. So we’re gonna hear updates on the state of collaboration, and I think this is gonna be, this is gonna involve some audience participation, I believe, Rich? Let me first introduce our panelists, though. Rich Platt is the Professor and Chair in the Department of Population Medicine at Harvard Medical School, and the Executive Director of the Harvard Pilgrim Health Care Institute. He’s the Principal Investigator of the Sentinel Coordinating Center, and as you heard, has been with this effort from its start, its first contract, as Mini-Sentinel at FDA. Marcus Wilson is President of HealthCore, which is a subsidiary of Anthem, and has also been a long-time leader in Sentinel and in real-world evidence development more generally.

Vinit Nair is the Practice Lead for Government Relations and Academic Partnerships at CHI, C-H-I, a wholly owned subsidiary of Humana, which has also been long involved in the Sentinel efforts. And Kenneth Sands, is Chief Epidemiologist and Patient Safety Officer at HCA, a large Sentinel partner, very much involved in these issues related to electronic medical record and other new types of data that you heard about in the last session. I am gonna turn this over to Rich, to lead the session and let’s get going. – Okay, great, if there’s a clicker? – There is a clicker. – I’ll take it, thanks. Okay, our thought was to spend this next period of time with you describing how Sentinel came to take the form that it currently has. What it’s like to operate it now, what the future would be like. Well, the future that at least FDA has asked us to consider.

And to do this as a series of three conversations. So I’m gonna use a few slides to sort of tee up the conversations, but the real critical content’s gonna come from my colleagues. And Mark, please jump in at every point. So let me just jump to the bottom line and say, from the very beginning, we have seen Sentinel as a piece of the very large interest in using real-world evidence to address questions that are of importance to the FDA. And let me talk about sort of six flavors of that. When we say Sentinel as sort of a short hand, most people think of that as the Sentinel data set, which is quite a valuable resource. But from the very beginning, it has been a condition that FDA imposed that it be possible to link back to original medical records. It is also important sometimes to link to registries, and we’ve put considerable effort into that.

We’re working quite hard now on the linkage to electronic health records. And then the activities that have come to the fore quite recently are bring able to link the Sentinel data set to patient-generated data and to test the use of the Sentinel infrastructure as a home for clinical trials. So in three parts, we’re gonna talk about how did we get where we’re going.

This is a slide from the 2009 Sentinel public workshop, that Janet Woodcock used, describing what FDA’s plans were for Sentinel. I would just say, we have been honored for the past years to be able to work with FDA, but the FDA drives the development of Sentinel. We build nothing that FDA doesn’t ask for. And we build it to FDA’s specs. So this was our original marching orders, and since there are a lot of words, I’ll sort of highlight some of them. We were instructed to create a coordinating center that could coordinate the work of a distributed system to work out what proved to be quite a sophisticated governance system to create capabilities for secure communications.

This says develop epidemiologic and statistical methods, but what it really means is adapting state-of-the-art methods to this environment, and then actually to evaluate the safety of medical products. So question number one is that nothing in what I just read said do it with claims, so why did we claims and not EHRs? And the answer is, well, at that time, EHRs were not very widely distributed in the US healthcare system. And so the only place to go for data on a sufficiently large population was to claims data. The second is an issue that is still important, which is, in the medical care system that we live in, if you wanna be confident that you know whether or not I have a myocardial infarction, or I’m hit by a bus while I’m biking to work, you can only really get that from my claims data, because the hospital where my electronic health record is may or may not take care of me, when I have my MI or encounter with a bus. And so that issue of coverage and knowing that the absence of an event really means it didn’t happen continues to be an important one.

And then the third important one is, there was a pretty good history of the use of claims for medication safety assessment. And we wanted to say we can build something useful to the agency and we have confidence that we know that at least the basic principles work. One of those examples was this work that four of the current Sentinel data partners, five of the current Sentinel data partners did together, looking at the safety of meningococcal conjugate vaccine. This was a study, so this was, we used distributed methods in five health plans. And for us, it proved proof-of-concept. It also proved that if you start from scratch, it takes four years to get an answer, it costs $7 million, and when an FDA reviewer said, this is an assessment of Guillain-Barre syndrome, can you also, by the way, look at another neurologic complication, the answer was, we’re sorry, we can’t do that.

This analysis was built for Guillain-Barre syndrome. So there were lots of things to improve on. And much of the development of Sentinel was intended to get at that. Why a distributed database, and not a single one? Well, originally, there was no single database that one could go to that was large enough for FDA’s needs. It is remarkable how large numbers get small fast, when you’re looking for medications that are not widely used in a population, when you’re looking at rare outcomes. And there were and still are important barriers to combining separate databases.

And I hope my colleagues will speak to that. There are good, and to my mind, sufficient reasons that talking about putting all the data together in a single data set would have meant that we would still be talking about the protections of that, and the uses of that data set. There are two other things that are products of having made that decision. The first is that a distributed system requires that you be extremely clear about what the analysis will be before you launch the analysis. Which means, among other things, there is a documentary record and you can’t really do data dredging, see a result that you aren’t so happy with, and so change the analysis after the fact for that. And then, the additional feature of having separate data systems that contribute to analysis provides this kind of assurance, sorry.

This is the result of an analysis that we supported, an analysis of FDA’s looking at the safety of oral contraceptives, and here’s the point estimate result from eight data partners. That single result is the combination of eight separate analyses. And it is, to my mind, very reassuring to see that in each of eight separate systems, the effect estimate is closely clustered around the final estimate.

And that provides a kind of reassurance, because there is so many decisions that go into creating the data set that is used, that this is a kind of guarantee that the system is actually answering the question that you care about. (audience member speaking faintly) These slides will be available. – Yeah, we’re gonna make all the slides available. So don’t worry about taking slide screenshots. – Yeah, so in our view, as we were designing this distributed system, and this was an activity that was jointly undertaken by our partners and by, with FDA, is we wanted the system to be able to accommodate many data holders’ data.

It was and is important to minimize the exchange of data. Information exchange should have no impediments. But the actual movement of individuals’ data should be minimized. The data partners should have maximum control over the uses of their data. We should be able to incorporate new kinds of data as they became available. But we have found that it’s critically important for people who understand how the data was created to be involved in designing studies and interpreting the data. Studies should be, it should be possible to implement an analysis identically and efficiently across the network. And we were anchored in the idea that we had to have a set of tools that were really reusable for this. My high-level summary is that FDA made three critical decisions at the outset that have been really important to Sentinel’s success.

The first is to call, to designate it as a public health program, rather than a research activity. Entirely appropriate, but it makes a big difference to the fact that this program operates under FDA’s public health authority, not as research that needs human subjects approvals, research approvals. Second is, out the outset of the, the understanding with all of our data partners was every query that FDA, every request that FDA makes is opt-in, you’re not obligated to any particular one of them. And I’ll be very interested to hear how you think about that. The lived experience is, it is extraordinarily unusual for one of our data partners to say, actually, we’re not gonna answer that query. And then finally, the fact that the distributed network really reduces the need to share confidential and proprietary data by orders of magnitude. So that’s my tee-up for the first of our conversations. And I’m hoping that, Ken, you might lead off in helping these folks think about what goes into deciding that it’s in the interests of an organization that has a day job to participate in this activity? – Sure, thanks, Rich.

And you know, the benefits for HCA being a partner in Sentinel is, as its most fundamental level, the indirect benefit, it’s the public health mission that underlies this effort. We see this as a really key national resource. And to the degree that HCA can be a partner in that, we’re honored and thrilled to do so. As long as the resource burden is reasonable, and I think that there has been deep thinking into the design of Sentinel in a way that that resource burden has been made minimal, relative to the advantages of participation. So I think that that has been the most fundamental thing, has been that indirect benefit. There’s other benefits. When I talk to our analytic group, they say they clearly have gotten better, and their skills have increased by virtue of being a partner in Sentinel. That that participation in data analytics and database design has made us stronger. And so that is a good reason for us to also continue to participate.

And I think that the test for that is that we use the data sets that are set up for Sentinel largely for internal purposes as well, because it’s our best, cleanest data set for doing queries. So I think that’s an important benefit as well. There is the cost, in terms of maintaining the program. And I think it’s great that we’re talking about the next generation of Sentinel, because keeping institutional interest I think will improve, as the capabilities of Sentinel improve. There hasn’t been a lot of down sides. The fact that it’s opt-in is a key element. But the reality is is that it’s extremely rare that some reputational issue or legal issue arises that would make us choose to opt out. – Marcus, Vinit? – Well, great comments, I echo everything Ken’s outlined, that the public health impact was really the original reason we did it, but there’s other things. You mentioned methods and being part of the methods development, as well as learning from the methods as they’re evolving and sort of being on the front edge of that has been really important. I think the other thing that I’d really want to say is critical, especially as we start thinking about, and the panel before talked about sort of the maturation of different types of data.

Rich outlined sort of building on the claims data, and as a starting point for ultimately getting much more complete view of the individual patients. Having data is, just having data and then doing analytics on the data is not good enough. We really have to learn how to use the data the right way. And learn what it’s good for and what it’s not good for. And I think that each data source, and each time we use it, and even different types of claims data, our sources as well as different types of data we’re adding to that, we have to really understand deep down what really drove its derivation in the first place.

And how did that impact our interpretation of the data? ‘Cause there’s a lot of variability in that. And more than anything, we want to get the right answer, not just an answer quickly, we want to get the right answer. That’s why better is at the front end of the better, faster, cheaper. You gotta get better right first. And then you can really focus on faster and cheaper. And I think that was really very much in mind of the construct of Sentinel in the first place, is really getting it set up so we can learn together. And being able to learn with the folks to my left, as well as many other folks in this room, and that are not here today, that are partners in all this.

I really look at this as being very much a collaborative shared learning environment. And Rich, you and your team at Harvard Pilgrim have done a phenomenal job, in terms of how you pulled that group together and kept us disciplined in that process. So it’s the learning process that’s really key for us in this whole process, ’cause we think that, there’s a lot of utility beyond just safety, and there’s no question, but we think we’ve gotta do it step at a time. – I’ll echo both your comments, certainly, on this. But I’ll try to add on. And I think it’s no secret all of the organizations have, like, Rich, you said, day jobs. And that’s not necessary a bad thing. ‘Cause what that does is, it is not, I think even Ken touched on this, this is not a key driver of our core business. And what that translates to is that we get to have the luxury to spend time on initiatives and not really get bogged down by the financials around it.

And calculating hours and dollars and cents. So a case in point is that we’ve been involved in this initiative, just like Marcus and team, for over 10 years here. And tell me, in corporate America, 10 years is a lifetime to maintain the systems, to maintain the teams, in an environment where folks can change. We all find different jobs and move on. Having the same core teams run the same operation for 10 years is a testament to the fact that of learning what Marcus what saying. The team has learned so much from Rich and his team, and FDA folks on this team, that they continuously feel a sense of professional accomplishment, enlightenment, learning, and that goes, that plays a key role for us as a partner.

Now on the side of the company, that’s a significant investment, like I said, for corporate America, for any company in corporate America to make for 10 years, to manage and maintain the same team. So we couldn’t be more excited to be a part of this. And I’ll also touch upon a little bit on what Marcus said, which is the importance of data. All of us have claims data, which forms the core functional system. But we are also adding on other layers of data.

Now keep in mind that health plans do not need all the elements that are made available for research for the purpose of paying claims. There are only certain elements that are needed. But all of us have made significant investments in gathering elements, for example, like laboratory values. There is probably no need for us to know exactly what the values are to pay a claim. But the fact that we are all invested in that, is the reason why we want to take that databases and use it for research purposes.

So these are kind of the some of the things that go beyond our typical desire to part of the Sentinel, but we do firmly believe that this is a national initiative. I would even go so far to say that it’s, to some of us, it’s also a patriotic initiative, to be a part of this exciting adventure here. – Mark? – I think this has been a great discussion, maybe I could just highlight some of the, what I’ve seen over the past decade is, just, and the last panel alluded to this as well, is just how much hard work has gone into this effort.

The notion of using data from real-world systems, that were collected for other purposes, paying claims, and now increasingly, lab results and electronic medical record information. And then using it for these initial safety surveillance activities, and a broader range of activities now. That takes a lot of work. It sounds really cool to use for data any analytics, but awful lot of data cleaning, standardization. You alluded to some of this, Rich. A lot of that work went into getting to this point and developing methods that are fit-for-purpose, for these data as well, as has been alluded to. I wonder if I could ask those of you on the panel, as you think about this better, faster, cheaper theme, now that there is a lot of experience here, a lot of well-established data models, well-established mechanisms for working together. How do you see that playing out in the future? Are there lessons that can be learned to extend Sentinel to some of the new types of data that we’ve talked about today, or to some of the new directions ahead? You may be wanting to cover this later in the panel, so I’m basically just teeing you up for the next set of issues.

But I did want to highlight that a lot of capital has been created by all of the work that’s gone in to getting Sentinel to the point where it is today, thanks to these efforts by all the data partners working in this shared governance as part of the network. – So let me show another few slides, which speak directly to that. One thing that I want to point your attention to, and that is the term data partners was not chosen lightly. What these people, and their colleagues, and their organizations are is not data vendors. And we are always mindful that this is a shared activity that we’re all in together. To your point, Mark, I say, so just sort the quick where we are, the term good enough for government work has taken on an entirely new meaning for us. And that is, at every step along the way, our colleagues at FDA have said, we intend to use this information to support regulatory decision-making that will affect the health of the entire nation.

And so it is essential that we understand that the inferences that come out of these analyses really represent the facts as they were. And that everything you do has to be auditable in a way that typical health services, or epidemiology research, is not. And so that’s been building that sort of robustness around this system has been an important, a very important consideration. And to be perfectly honest, it’s added a lot of costs to the system, and from my perspective, it’s been money and effort well spent.

So Sentinel today is one that is in routine use, as Gerald, and Steve, and then Greg said. It’s gotten some public recognition, as being a high-quality and useful resource for the nation. The data partners look like this, those who have attended this meeting in the past will notice that, for the first time, CMS data is part of the system. That is where Duke Medicine is now above the line as the data and scientific partner.

That represents the fact that Duke is now serving as the focal point for a CMS node. And so we now have access to, thanks to the considerable efforts of our colleagues at FDA, access to a 100% sample of the CMS population. We spent a great deal of time developing a data model that informaticians say a simple data model. And it is, the big advantage of this simple data model is that all of the data partners work together to agree that they could faithfully populate this model in a way that meant the same thing from data partner to data partner.

I don’t actually know what’s inside your systems, but my understanding is that the term ambulatory visit is not a defined term, but there are several hundred kinds of encounters that occur when somebody isn’t hospitalized, but they need to be rolled up into ambulatory visit. And so there was discussions that went on for quite some time. as Vinit pointed out, when it became clear that laboratory results would be available, we could add a data table. As Hospital Corporation of America joined, we added inpatient pharmacy data and transfusion data to be able to support those kinds of studies. So it is proven to be, I’d liken to, let’s say, a Land Rover. It’s, by design, robust and simple, but can serve as a platform that can take FDA in a lot of directions. FDA made the decision that it was worth investing quite a lot in curating the data, so that the data are, essentially, always ready to be queried. Each data partner refreshes its data approximately quarterly. It is usual that in that refresh process, something in the 1,400 data checks that are performed arises that requires a conversation between our team at the Operations Center and the teams at each of the sites.

And in approximately half of those data refreshes, it’s necessary to rerun the refresh to deal with that data issue. It emphasizes, again, the importance of having people understand the providence of the data involved in creating it. These data, this snapshot of what’s in the data set, have been updated since the commissioner’s numbers, because it shows the Medicare fee-for-service data, showing that we now have quite a substantial fraction of the US population under direct observation. Among people currently accruing data, it’s about 67 million people. So Sentinel has become more and more robust now because of our ability to add important new data sources. For the past several years, we have shown a slide like this, which was originally drawn by our colleagues at FDA saying that Sentinel provides tools for the toolbox.

What this slide doesn’t really convey is how much more sophisticated these tools are now than they were when we first showed this slide. The main cohort identification and descriptive analysis tool is now in release 5.2.1, and has become a workhorse for a whole variety of studies. We have been committed to making these tools available to the public. The next of the public trainings in what these tools do is occurring tomorrow and will be available online. But it’s now possible to do whole studies that essentially replicate the kind of four-year, $7 million study that I showed originally, in a matter of months using existing tools. When we say months, most of those months go to exchange between FDA scientists and Sentinel Operations Systems scientists, to actually design at the specifications to a level of detail that they can actually be implemented.

The increasing sophistication of these tools is in direct response to the needs of FDA scientists. I want to point out to you that among the things that is now being bolted into the tool set is the ability to link mothers and infants. You could say, why should that be a difficult thing to do? And all I could tell you is that one person goes into the hospital, and two people leave the hospital. And that second person only keeps the mother’s ID for a very short period of time, and then has a whole new ID and it’s real work to understand how to link them and how well to link them, and how confident we should be. So using these tools, we now have a suite of tools that can take agency scientists from doing very simple feasibility queries very quickly to doing complex kinds of descriptive analyses, to doing full inferential studies.

Identify a cohort of new users of Drug A, and a cohort of new users of Drug B and compare their outcomes in ways that adjust for a whole variety of confounding. I’m not gonna take us through an example, because that’s not the point of today’s discussion. But it is important to note that along the way, it has become necessary to build in capabilities to do things like understand how soon after a drug exposure occurs is a person really at risk for the outcome that FDA cares about? What do you do about the fact that sometimes people refill their medication prescription before the original one was exhausted? What do you do about the fact that sometimes there’s a gap between the end of one dispensing and the new refill. A whole series of questions, and those are now capabilities that are built into these tools. And the answer depends on the question under evaluation. And we’re finding that it’s sort of, with every set of questions, there’s often a new set of capabilities. And to the greatest extent possible, we update these tools.

The commissioner mentioned that there have been 254 analyses done in the past couple of years. We don’t have even a back of the envelope estimate for what would it have cost to do these analyses using the old kind of system. But the answer, frankly, is more than the agency could afford. And so many of these are analyses that never would have been done, absent this. I said that an initial boundary condition for the agency was that it be possible to link back to the systems of care from which these records were generated.

And we’d say that there are three kinds that we’ll talk about right now. The first is this expert knowledge that only lives with the people who are in the systems. The second is the ability to deal with these data quality issues that arise every time you look at a data set. And the third is the ability to go back to obtain full-text provider records when they’re needed. And there are two major reasons to do that. One is to validate the ability to use coded data. And that coded data could be coded data in the EHRs as well as in claims. And second is the example that Steve Anderson described.

Did this baby have intussusception? And to be able to read that record and understand that. Now we had teed up another conversation here, and I just need to test you guys. I realize we want to leave time to have a really in-depth conversation about where we’re going, so do you want to weigh in here on how it’s going, or should we keep on moving? (man speaking faintly) Maybe brief comments. – I think it’s going pretty well, so let’s just keep it. – Well, that was a brief comment. – So where are we headed? And I’ll say again, we only go where FDA says we should go. But among the things that are extremely clear is that Sentinel needs to be able to take advantage of EHR data, which now is increasingly available. That it’ll be important to use methods that are appropriate for electronic health data. And that includes things like using machine learning to develop algorithms. And to understand how to use natural language processing for use in free-text information. Now in order for EHRs to be useful for Sentinel, we have to recognize that in a lot of ways, we are now, with regard to electronic health records, we are almost to the point where Sentinel was with regard to claims 10 years ago.

That is, there are lots of EHRs now, and no EHR system is sufficiently large to be able to support Sentinel’s needs. The second is that, to the best of our knowledge, there is still no standard widely implementable method for taking advantage of the information in a bunch of separate EHR systems. That is distributed machine learning and distributed natural language processing is a topic that’s gonna need serious thought. And finally, it is going to be necessary to link EHRs to claims for many of the things that FDA is interested in. The kind of data that Hospital Corporation of America has, complete data for inpatient stays for 5% of the US population is ideal when the exposure and the outcome occur in the hospital. But for many, for the large majority of questions that FDA has, it’s going to essential to be able to link the information in the EHRs with the claims data, for the same reason that we described about me and the bus.

That the EHR data by itself is ordinarily not gonna make it. One of the things we didn’t talk about, that I think would be helpful, if you folks will discuss, is what it takes to link data. And in particular, the governance issues surrounding who gives whom what data, and what kinds of assurances do there need to be to make that happen. Because I know that Marcus and Vinit, you’re actively doing that as part of PCORnet activities. Part of my takeaway is we will need to figure out ways to do distributed analysis of, figure out that this is Richard Platt’s EHR here, and Richard Platt’s claims data here.

Never actually put them together, but to do distributed analysis that uses both of those. And this afternoon, my colleague, Darren Toh, will talk about some of the methodologic work that he’s been doing to give Sentinel that capability. Bob Ball in the Center for Drugs is leading activities that are aimed directly at these set of issues around EHRs. We have a work group that is assessing existing best practices to be able to automate the review of medical records. And Bob and Jeff Brown are leading the development of a workshop, I believe Duke-Margolis is deeply involved in that, too, to bring the best and the brightest together to help inform the development of an agenda for that. So here’s the second part of the reason that it is so advantageous for Sentinel to be built on the foundation of live systems.

And that is, these are systems that are able to go back, not just to medical records, not just to hospital systems, but to directly engage their patients, their members, and providers. So I’m gonna take a minute to talk about the work that FDA-Catalyst is leading. So FDA created the FDA-Catalyst program to deal with situations in which the Sentinel Initiative would engage directly with members or providers. And to do that in a way that takes advantage of this infrastructure. David Martin led the development of a mobile app that studies safety and effectiveness. Mobile apps are widely appreciated to be useful adjuncts for this. The goal here is to develop a mobile app that works for Sentinel. That is the special sauce here, is to be able to directly link the information, that a, use Sentinel to identify people who have conditions of interest, and recruit them to provide information, which is then directly linkable to their information in the Sentinel distributed data set.

So that’s the special sauce here, and why it took considerable time and energy to do that. The first proof-of-principle has been completed. This was a study where pregnant women were recruited. The patients were directly involved in developing the design of this app. The goal was to test this with 50 women. So 1,000 women were approached, more than 50 agreed to participate. And the results have been quite positive. It proved that women were willing to provide information two or three times a week. The information that they provided was quite useful. And it gives us real reason to think that it is now possible to use this system to address questions that are of interest to the agency. With the Center for Biologics, we’re developing sort of a next set of options around populations that for whom it would be of great interest to collect information. And part of the reason for that is that so much of what’s of interest is not in claims, and it’s not in the EHR, but it could be obtained directly from those individuals.

And then finally in clinical trials, Gerald mentioned the IMPACT-AFib study, my colleague Chris Granger will talk about it in more detail this afternoon. What he won’t say is the smart idea to choose this as the topic for our randomized trial was his, and he deserves a lot of credit for bringing this to us. So IMPACT, so you know we’re dealing with cardiologists when you have an acronym like IMPACT. But this is an 80,000 person, individually randomized trial involving Marcus’s organization, and Vinit’s organization, and three others, in which we’re testing the ability to motivate people who have a condition to be the change agents with their clinicians. So here’s who’s involved. As always, we involve a patient representative is designing the study. And there is an intervention aimed at patients and one at their providers, and the goal is to try to activate patients, to activate their providers to initiate oral anticoagulation for individuals who have atrial fibrillation, have other risk factors for stroke, and who are not getting anticoagulants.

The important thing to say here is the entire infrastructure for this study is available in the Sentinel data set. You can identify the people who are at risk, you know who their providers are, the health plans directly engage with their members. And then, when the study is over, all of the results will come from the Sentinel distributed data set. So this study is, an 80,000 member study is never done for less than something like $80 million.

And this is a study that is a $5 million study. Finally, I’ll say that from the very beginning, FDA has said that it’s important for Sentinel to be a national resource. And I just want to say that they work that we have said was coming along has been doing quite well. The IMEDS program now has a portfolio of research projects, where regulated industry is working with Sentinel data partners. The NIH Collaboratory Distributed Research Network is making steady progress in providing FDA scientists the ability to engage these data partners. And the partnership with PCORnet is quite robust. Okay, so we’re now to what I think is the most interesting part of the discussion, which is, so what do you think about how we’re gonna make this sort of additional leap, in terms of expanding the uses of the system we have? And what do you see as the big challenges? Marcus, do you wanna lead us off? – Sure, sure, so I think we have made a lot of progress in 10 years, and just to kind of go back to one of the topics before, because within those slides, there’s a lot of important points, also a lot of questions.

And we’ve talked a lot about electronic health record data, and the understanding that a couple of things in EHR data. So first of all, we have to master the use of those types of data in combination with claims data and other sources of data, including genomics data, at some point, as an important point. But those of you who have worked with EHR data understand just how fragmented it is. Also, how inconsistent it is. And so the learning curve that we had for claims data, and you talked a little bit about, you made the statement that we’re starting with EHR sort of where we were with claims data. I would say we’re a lot further behind on EHR data, because we were a lot further ahead, in terms of standardizing how claims data was captured.

And it was a bit more uniform, even though a lot of the things behind the data can confound the claims data in non-apparent ways, which is why we did the distributed model for Sentinel in the first place. There’s so many things in EHR data that impact the interpretation of the data, the meaning of the data, that that’s critical to get right. So there’s a huge learning curve in this. And so I think that’s part of it. The other part you mentioned was the fragmentation of the data. So you mentioned, if Rich Platt’s data was in the hospital and in a claims system, but it’s also in a primary care practice, a specialist the street, an ambulatory care center, there’s all different types of places where your fragments of your data, your EHR data exist. And what we learned 20-plus years ago, when we started HealthCore, we went out on this mission to try to create that view, so that we could do incremental research and insight development, and what we learned very quickly is that you have to, there’s one place you have to start.

‘Cause we tried starting with the chart, we tried to do it within the outpatient setting, we tried in the hospital setting, we tried all different places to start building that view. And what we kind of came back to was no matter how we worked from different vantage points, we had to start with the claims data, because of the things you talked about. It serves as the index. It’s imperfect in many ways, but it’s the index, and the best index we have, even in its incompleteness, of starting this. And so then you can start building on top of that.

And I think so that’s where the strength that we have in the foundation with Sentinel today, but we have to aggressively now begin to look at bringing in the other data, and learning to use it on top of the data that we have to date. Because there’s so much value in its creation. So I applaud a lot of the work that’s going on in that space, and also encourage us to continue to stay disciplined as we do it, to make sure we’re really getting it right a piece at a time, because what we’re building is a national resource and we want to get it right. – So Vinit, will you say something about what was involved in your engaging in this clinical trial? Because you have tens of thousands of people, of your members. Obviously, sending them a letter saying, this is your health plan, you might have a problem, is not the kind of thing you do lightly. – So certainly, Rich.

And so we’ve been constantly trying to balance between what I call as thirst for research versus risk. And sending letters out 50,000 individuals becomes a part of that. Because, keep in mind that this population, as you all know, it’s not a sanitized population. They receive a whole lot of mailers from health plans on all sorts of things. And let’s face it, what happens to most of the mailers that you get from your health plan? It goes to the trash can, unless there’s a check on the outside. So getting people’s attention, in terms of having this as an incredible important part of the study, is one key challenge.

But going beyond that is also, I think I’ll touch upon something that Marcus said. It is the value of bringing EHR datas into the space. Data is not perfect. We are still learning, a long way to go in terms of learning that piece. But I’ll tell you one piece, and I’ll weave this in, too, by saying something we have said since day one here, which is, and not to downplay the importance of the technical stuff, and that you have heard these things all the time.

Technical is great, governance is even more important. And where that falls into this is, there is even higher level of, what do you call, concern and care we have for the data, for EHR, especially because of the sensitive elements that the data contains. Now somebody once told this to me at some meeting, and I want to give that person credit. Claims and EHRs are stories of people with their details removed, absolutely true. And I think that’s a level of sensitivity as an organization we keep towards the data.

That certainly adds to all complexities, like Marcus said, another layer of protecting the data. And with all the protections come, it gets much, much harder to use it. So how do we find that right balance of being able to use the data, at the same time protecting the data. And then, like Rich said, sending these mails out as individuals, and let’s face it, this data is not accurate, like we all said. So people do get mail sometimes when they do not have that particular condition. So how do we handle those kind of concerns? So it has been a case by case basis. So far, so to speak. Our engagement in IMPACT-AF has been, thankfully, great so far. We have not hit any, we have not done anything stupid.

So it’s been going great. But so far, we’ve been handling it on a case by case basis, making sure that the information’s accurate. It’s a learning process right now. – So Ken, I know Hospital Corporation of America has experience doing cluster randomized trials, which is all the rage in some circles. So could you give a sense of is it a real thing? Can organizations like yours do them? And do they have a future to contribute to comparative effectiveness assessments? – Yes, I think absolutely. We have 180 hospitals that can participate in trials.

We’ve run trials with as many as 140 hospitals participating simultaneously in a single cluster randomized trial. And I think it’s a highly efficient way to look at an intervention. Right now, in partnership with Harvard, we’re doing the Swap Out trial, which is comparing two therapeutics, mupirocin versus iodophor for MRSA decolonization therapy. And we have randomized, by hospital, and are collecting that data, which, again, there’s resources involved, but I think that the cluster randomized model will allow us to get a lot of data at a relatively low cost. We find that our hospitals enjoy participating in these trials. It’s a sense of, it’s extra work, to a certain degree, but it’s something that engages our nurses, engages our physicians, and so we see benefit from that, aside from the fact that we benefit from the information achieved.

As I looked at your last set of slides, it’s an ambitious agenda. And we talked earlier about the balance between resources and benefit. It looks like a more ambitious, and a potentially more resource intensive agenda, but it strikes me as having a higher degree of overlap with our internal priorities. We’re spending an enormous amount of time thinking now about how we can use our EHR for natural language processing, for machine learning. We are trying to set up our EHR to be able to have those capabilities.

We’re a fair way in that direction, but we’re not all the way there. But it’s an area where groupthink would seem a really powerful way to move things more quickly. – Can I add onto that? Just the point about the cluster randomized trials, and it goes back to designs. There’s only so far, in many cases, retrospective analysis of data, of any types of data, regardless of how robust, it can answer certain questions, but in many cases, there’s a limit in what it can answer. So prospective designs are critical. And I think the prospective real-world designs, like pragmatic trials and large simple trials, those types of things for me, ultimately, are gonna be essential. And advancing the methods, advancing the creation of that environment to do this are gonna be absolutely critical. And I actually think that Sentinel, in this case, provides a great foundation for that. Because you can do the analysis, and then quickly move from that to a prospective design, and on those populations, because you’ve done a lot of that work already. So being able to do the prospective designs on the back of something that we did in a retrospective fashion, if you would, I think, in my mind, is a great potential opportunity for us.

– Great, Mark, do you wanna take over? – There’s just so many ways to go forward in this very strong Sentinel foundation that you all have collectively built. I would like to also see if there are any questions from the audience. Maybe this would be a good time for people to let us know about that. I just wanted to make one comment, too, about coming back to this theme of better, faster, cheaper, and Ken’s point reminded me that that’s a mantra that the rest of the healthcare system is facing today as well. And with these efforts on getting towards more real time data, integrating more sources, it really seems like there’s a potential for much more alignment with a lot of other strategic interests of the data partners around demonstrating that care can be delivered more effectively, targeted to the right patients, more of an era of precision-oriented medicine that uses evidence-based best practices.

The understanding of these data systems, bringing them together, and having analytics to support that, is not only relevant for the next round of questions from the FDA, but I think may be increasingly aligned with healthcare systems and with data partners that themselves are under the gun for demonstrating better outcomes, lower costs, fewer complications for the patients that they serve. And I don’t know if you are, I see some head-nodding, I don’t know if there could be any more comments on that. But I did have a couple of questions that I’d like to ask.

But let me see if there are any questions on that, and then I wanna go to the audience. And then I might have a followup, if there’s time. – And I need two seconds to– – I know you need to end it. – Then I’ll just do a quick comment on that, Mark. I think it’s an excellent point. The health plans, in our world, we spend a lot of time trying to close gaps in care where we know what to do and trying to get folks to do it.

The IMPACT-AF’s a great example of that. At the same time, and we know what interventions are actually gonna work best, so in that case, it’s trying to get the providers and the patients themselves to a point where they’re gonna achieve the best outcomes. The other part of this we can’t ignore is that in most cases, and Rob Califf points this out a lot in his talks out there, is that most of the time, we don’t know what to do. And so it’s really critical for us to figure out what to do, especially on the individual basis, or in very specific populations, the evidence is not granular enough to really have, to make the best decisions. So we really have to look at a broad array of evidence development and building the resource.

The resources and the environment to do that are really critical for us. – Yeah, I do see Sentinel contributing more and more into that overall effort as this alignment continues. Please, and we’ll have to keep these questions quick, so much to cover. – Hi, Suanna Bruinooge with the American Society of Clinical Oncology. I’m wondering, we have a lot of interest in oral adherence in the cancer therapeutics field, and I’m just curious because you’re HCA, and CHI, you clearly have a lot of reach into pharmacy data, provider data, and probably stuff that our members don’t have, quite frankly. So I just wanted to hear you talk a little bit about maybe access to pharmacy data. Do you work with PBMs to get that? Or kind of what you’re capability is in that sort of ambulatory space as well. – It’s probably worth pointing out that Sentinel works with actual dispensing data, whereas EHRs have prescribing data.

And there’s a big difference. – And you’re right, and in fact, I would say that, so if you’re looking at all the kinds of different data, from ambulatory to the inpatient to, let’s say, a pharmacy. I would certainly argue that pharmacy is probably the clearest line of data that we would have. And that’s not because of anything else, but other than the transactional nature of the data is that the waterfall is much, much more clearer. So to answer your question, yes, we do have access to that. And in fact, that’s probably the clearest data system that exists within our parameters right now. – Yes, same here. – Please go ahead. – George Plopper, Booz Allen Hamilton. I was wondering with respect to recruiting partners here, how you address the idea that you’ve got different EHR platforms that you’re trying to merge here.

I’m wondering how agnostic Sentinel is with respect to that. And any challenges that you’re encountering marrying those data together. – Yeah, thanks, and that probably is going to be the last question we have time for in this session. But Rich, you were saying this is sort of like claims 10 years ago? – Honestly, that is an enormously challenging question. The EHR landscape is highly vulcanized. The overlap between, we’ve said you have to have claims, and the overlap between the EHR systems and the claims systems is only partial.

That is if Sentinel now has, looks at 20% of the US population, and the largest EHR systems look at 20% of the population, you got way under 20%, which is problematic, so it’s among the many, many challenges that were. To go from what is obviously a good idea to actually being able to stand up something that can work effectively to support the agency’s needs is, on the one hand, the day-to-day work is not glamorous. My staff made me take out a whole bunch of slides that I wanted to show about what the heck goes on to just take a straightforward query and turn it into a straightforward answer. And they said, you can’t do that. But I’m hoping we’ll come back in a couple of years and say, look, it looks pretty simple.

And we’ll have succeeded if it looks pretty simple, even though it isn’t. – Yeah, so that’s certainly ahead. The other questions that I wanted to ask pertain to this notion that came up earlier of moving from signal strengthening and confirmation issues towards signal detection issues. Which it could be done with all this capital investment, but means going to a different model, running potentially a lot more associations. That’s something that may be in the future with more of an emphasis on effectiveness determination and randomization. Sentinel’s had to deal with issues of consent that were not part of the earlier process, but may be a bigger part in the future. We don’t really have time to talk about those now, but I do wanna turn back to you for, Rich, any final comments. – Well, the final comment is you see four of us here. There are 400 who are actually doing this work. Some of you are in the room. So folks, if you’re part of the Sentinel team, either in the Operations Center or the data partners, just stand up for a second please.

(audience clapping) – I have some online, too. – And then, I just want to show you sort of all of the people who make Sentinel successful. – This gets longer every year. – It sure does, so Sentinel only works because there are a lot of people who have great expertise that they’re willing to contribute to the system. So thank you. – All right, well, all of you, Rich, Marcus, Vinit, Ken, thank you for contributing your expertise, too, we really appreciate you’re joining us for this panel and all of the work that’s going on, and the opportunities for the future. Thank you. (audience clapping) All right, we have covered a lot of ground this morning. I’m trying to stay just about on time, so we’re gonna take a short break now, and reconvene at 11:15, lots more to come.

Look for to our next session. – Thank you. – Thank you, very good. (people chatting) – Okay, we’re gonna go ahead and get started in our next session. If I could ask you all to please start taking your seats. Oh, sure, and I’ll ask out panelists for the next discussion to start making your way up to the stage, thank you. How’s it going? Okay, welcome back, everybody, my name is Greg Daniel, I’m Deputy Director in the Duke-Margolis Center for Health Policy. Let me echo Mark’s earlier welcome to all of you for joining us in the Sentinel annual meeting. In this next session, we’re gonna take a bit of a deeper dive into how the FDA is utilizing Sentinel within its own processes.

This panel with start with presentations from both CDER and CBER Sentinel program leads, and then we’ll pivot to some reactions from two panelists that are taking quite of a unique perspective. One is from the FDA’s Office of New Drugs, so that we can hear how that group is incorporating the Sentinel results and processes within its decision making, as well as a reaction from the private pharmaceutical industry. So joining on this panel, we’re gonna start off with two presentations from FDA, Michael Nguyen is Sentinel Lead at the Center for Drugs Evaluation and Research, so the drug center. Azadeh Shoaibi is Sentinel Lead in the Center for Biologics Evaluation and Research.

And then following those two presentations, we’ll have some reactions from Brian Bradbury, Executive Director of Amgen’s Center for Observational Research. And then also, Mwango Kashoki, who is the Associate Director for Safety in the Office of New Drugs at CDER as well. So I’m gonna go ahead and turn things over to Michael, thank you. (speakers speaking faintly) That’s all right. (Mwango speaking faintly) – And they’re like oh my god. – All right, so my name’s Michael Nguyen, I’ll be speaking on behalf of a lot of people in the Center for Drugs, who make this all possible. Many, many people from the Office of New Drugs, as Greg mentioned earlier, the Divisions of Epidemiology at CBER, the Division of Biostatistics, and many, many more. So we’re gonna take a little bit of a deeper dive into this. So I’m gonna talk about three things in my talk. We’re gonna talk about measuring the progress through the lens of independent assessments. Then we’re gonna review the system that we’ve created and how we’re using it at CDER.

And why having a system matters. So let’s start here. So these are independent assessments done by a highly regarded consulting firm, as part of our meeting PDUFA commitments. There was two done, one that was completed in September 2015, which was the interim assessment, and the follow on, which was completed in 2017. So why mention these at all? Because they provide a longitudinal perspective. The 2015 report covered that both the Mini-Sentinel period, as well as a year into after Mini-Sentinel. And the 2017 brings into the much more recent period. And into after Sentinel’s activation. And it was a consistent rigorous methodology that was done independently, so you don’t have to believe what I say, but these are words and conclusions drawn from an independent entity, using methods such as interviews with FDA staff, Sentinel Operations Center and data partners. They implemented an electronic survey for those, cover the gaps where they weren’t covered by the dozens of interviews. They reviewed key operational metrics that we provided them. Review of publications and reports. And most importantly, they measured the program against predefined qualitative criteria, in what they call the Sentinel Maturity Model.

So here’s, at the highest level, the interim assessment results. You can see here these are three of the dimensions that they evaluated, talent and organization within FDA, mostly the management of it. Governance and process, so you can see the grade us on a qualitative scale of zero to four with four being the full maturity. And I’ll focus entirely on the beige bars here, where you can see our scores, which, in 2015, showed a lot of room for improvement.

So to give you more flavor of just how far we had to go, here were their recommendations in the end of that report, to give you a sense of how much we had to go, in terms of going forward. So they were saying back in 2015, by 2017, we need to really aspire to increase user participation in Sentinel. We needed to really broaden access to the full set of modular programs to our staff. And here, they emphasized particularly at CDER. We needed to introduce much more systematic and consistent processes for triggering the Sentinel System. We needed far greater transparency than what we had. And then you can see here, they even stepped down into 2020, and said, you know, in five years, or four years later, try to aim, try to aspire to take advantage of the high-level customization capabilities that were already in Sentinel.

And then by 2020, they also said, look, FDA has invested an enormous amount of resources in creating this system, the goal is to regularly use it by 2020. So there is much to report, I guess, in progress in two years. You can see here, in the 2017 report, just how much has changed with substantial investment in our leadership. You now have statements where Sentinel has been widely accepted within FDA as a useful regulatory decision-making tool for safety issues. In fact, I’ll go deeper into that and say, all 16 Office of New Drug Divisions responsible for regulating prescribed drugs are involved in Sentinel in one way or the other, and had been involved. The awareness of its capabilities among the staff has materially increased.

It’s been incorporated into the regulatory process, its results have been instrumental in deciding a number of critical public health issues. The infrastructure and associate suite of tools are increasingly robust. And the last statement here is really testament to just how much has changed. The question is now no longer if Sentinel will be used in regulatory decision-making, but rather how best to cost-effectively scale and embed it even further. So even though things looked fairly dire in 2015, I think we turned it around in a couple of years in 2017.

And so here’s those six metrics I spoke about. Process, governance, talent and organization, methods, analytic tools and technology, and strategy and value. Some of the bottom three weren’t measured with numbers until the second assessment, but you can see now that the program has reached what they determined to be a high level of maturity. As we got the results back, I did probe a little bit, and I said, well, can you give me an example of a government agency or entity that has reached full maturity? Thinking in the back of my head, there’s a lot of respected agencies around, Centers for Disease Control, it can intervene in Ebola outbreaks, NIH can have breakthroughs in science, and maybe US Special Forces, Seal Team Six, like, has anyone actually made it? And I’m still waiting for that response, so. (audience chuckling) But more seriously, no organization worth its salt would accept a marking of four as full maturity. We would always strive better, so I’ve made peace with the results. But I mention these, because again, it’s an intimate assessment, it’s longitudinal view, and you can go back, and see, and get more detail of just how we did.

All right, so let’s review the system we’ve created and how we’re using it in CDER. I have to go back to FDAAA, because that was the originating legislation. And there are two key parts to this, which is Section 905 here, that mandated the creation of ARIA or Sentinel. And ARIA stands for Active Risk Identification and Analysis system, that’s right out of the legislation here, highlighted in red. And at the same time, FDAAA gave us, the first time the authority to require post-market required studies. And so Section 905 and Section 901 are balanced on this paragraph right here, which basically says, we have to consider the sufficiency of ARIA before issuing a PMR. And if ARIA’s sufficient, we cannot issue a PMR. And so what is this sufficiency? It’s having adequate data, having appropriate methods to answer the regulatory question of interest to lead to a satisfactory level of precision. There’s all kinds of epi-terminologies behind this, but this is at the highest level what ARIA sufficiency really is about. And so to that, we’ve created a range of analysis tools here.

You can see summary tables, which produces simple code counts, Level 1 analyses, which produce descriptive analyses on adjusted rates. Level 2 analyses, which produces adjusted rates with sophisticated confounding control. These are your epi studies, that your used to seeing in JAMA, and New England Journal, and Epidemiology, and PDS. And then there’s this Level 3, where you take those Level 2, and you make it sequential. You look at risk assessments over time. So it’s really about the tools, ARIA’s really made up of the tools and the electronic data in the common data model. So how are we integrating this? The assessment basically concluded that we’ve done a decent job integrating it. And this is one of the examples, where ARIA is now being integrated fully into the approval letter. You can see here the intent to study a safety issue in the ARIA System. Paragraph one is just, it invokes FDAAA. Paragraph two says, look, we are now studying septal perforation, cataracts, and glaucoma in ARIA. And number three is a paragraph about transparency, and when are you gonna see the analysis parameters? When are you gonna see the study design, and then when are you gonna see the results? All built into the approval letter, now all public.

So what have we done to date? Rich has talked about how we’ve done 254 analyses to date. Here it is over time, just by quarter. The bottom boxes in blue are the summary tables. The purples are the Level 1s, and the tops are the Level 2s. And so there’s two flavors of analyses here that you see. First are the more simple things that you can end in a Level 1. And the second thing is that the package, the bundle of analyses required that include feasibility analysis as well as the inferential analyses that fully answer a safety issue.

So it’s not a one-to-one correlation, it’s not like we’ve evaluated 254 safety issues. It’s the bundle to answer, required to answer it. All right, so we’ve become much more transparent. And we’re trying to get more transparent. We’ve launched a new page called How ARIA Analyses Have Been Used by FDA. And I just want to make a couple of points here. You can see just the wide variety of ways in which we’re using it. Everything from drug safety label changes, which you would hope, here’s the new warnings and precautions, helping us respond to citizen’s petitions, presentations, and advisory committee meetings, helping us just do background disease surveillance to inform product development, of novel RSV therapeutics, et cetera, et cetera. And the other thing to note about this page though is that in no one of these did Sentinel play the defining role. That was never the goal of the system.

We’ve said since the very beginning, 2009, that Sentinel was not meant to replace any system, but to augment the strengths that FDA already had. And so the bar here is that, was Sentinel data considered as part of the regulatory decision? Did it make a meaningful contribution? And that’s what gets you here. The other point to see in this list is that there’s a natural delay. So even though we have 254 analyses done, it takes time for the analysis to be done, it takes time for FDA to incorporate in the decision-making, and then it takes time for us to clear and get these online.

So expect more to come. All right, so let’s take a look at ARIA sufficiency then. If you look at all the serious safety issues evaluated in ARIA, this is by sufficiency and year. And this, I should caveat and say this is preliminary data. We’ve had 89 safety issues go through the system since 2016 to 2017. You can see the relative, in blue, which were deemed not sufficient to study, and which were deemed sufficient to study. And you can see the relative proportions. And overall, 49% sufficiency rate is not bad. I should also say that these are questions where FDA thought it was possible to do in an observational design.

So if we had decided already up front that only a clinical trial could answer the question, that’s not counted here. But for those subset of questions where we thought this could be done in an observational study, getting it at about half is about right. If you think about the FDAAA’s originating legislation, where it said FDA can issue and force manufacturers to do a post-market study, but at the same time, we have to strengthen our abilities. And so that balancing on the fulcrum of sufficiency, 49%’s not bad. All right, so here’s the same set of 82, but now divided pre-approval and post-approval. And you can see here, pre-approval, the sufficiency rate’s less. And I think that’s to be expected, whereas the post-approval, it’s higher. And why is that? Some of the reason here is that size of the safety database is much smaller, pre-approval. And so the questions are much tougher. And the questions are much wider ranging. They’re primarily driven by the therapeutic mechanism of action, and some small imbalances in the clinical trials that may or may not be due to chance.

And so you get this lengthy list of possible questions. Whereas post-approval, it’s usually driven by single-signal. And it’s well-defined, it comes out of, often, observational data, and so following up with an observational study makes sense. So why do we keep all these metrics? Because it helps us develop the system. These are the reasons why we deemed ARIA to be insufficient in order. And you can see the first four are all because of lack of data, whereas only the last is because we didn’t have a lack of a tool. So the fixing a tool is easy, it’s writing a fast code and getting it to work. Fixing the data is much larger problem. And Bob Ball will be leading a panel on that later this afternoon on it, so I don’t want to steal his thunder. All right, so as I said before, keeping track of this informs development, and here’s just a couple of select examples. We had questions where, look, there was no mother infant linkage and no analysis tool.

And so we created that linkage, and it’s anticipated, and it’s coming this October. There’s other questions where, look, even with the size of the Sentinel database, it still wasn’t enough, because it was all under 65, or primarily under 65. And it’s the elderly population that uses the drugs. So we integrated Medicare data, and that just went live. Incomplete cause of death, we’re trying to facilitate linkage with the NDI. And then performance of claims for outcome validation, as I said before. And there’s a range of projects there, and those are ongoing. So keeping the statistics help us make us better. The transparency and access are important, one of our main constituencies is industry. So how do we make sure that this is transparent? And CDER notifies all sponsors, and is committed to doing that.

And we post all Level 2 inferential analyses. And so we will notify, for at least the pre-licensure, we will notify you within the approval letter. Going forward in post-licensure, we will send a custom letter to you the moment that we have a final study design in place. We will list those L2 analyses on an ongoing assessment page, which is live and you can see. We post, when they’re available, all the analysis parameters, all the results. And so if you wish to, you can then take that analytic package literally and go to IMEDS and run in the system.

And change the parameters that you think need to be changed. So we will continue to support IMEDS, to enable, as I said, Sentinel infrastructure to external entities, such as industry. Additionally, what I learned this year is that the single most important thing we could do to help open up Sentinel was to create a publicly available synthetic data set. And that’s what we’re doing. And fully worked examples so that those in the community, especially academia, can see exactly how this works, and deconstruct it and pull it apart. We are also doing public training on tools and methods. CBER mentioned a whole bunch of public trainings, we’re doing one tomorrow. So we are committed to this. So lastly, just I’ll have a last section, why having a Sentinel System matters. So I want to take you back, to what it’s like to launch a multisite study. And the meningococcal vaccine study was a great example that Rich brought up. But here’s what it takes. You have to identify and prioritize the issue. You have to find the money within your agency and allocate the resources, and make an organizational investment decision.

Then you have to do the contracting. You have to issue the RFP, you have to then wrestle with the contracting office to show that the choice you made, which may not be the lowest cost bidder, is the rational decision and the best for the FDA. You award it, and then the work starts, after all of that. You have the study design, you do your protocol, you do the statistical analysis plan. Then you have to extract and QC all of that data from however many data partners contribute to that study. Then you custom write the code, and you write it, and you QC that code. And then you begin the feasibility analysis. And when you do the feasibility analysis, it often changes he code that you wrote, it often changes the study design you wrote.

And only after that do you finally do the inferential analysis. It’s a huge lift, and when you do it as a one-off, none of this gets reused. None of this gets reused. So let’s talk about doing a multisite, multi study within a system now. So you take away several of these boxes. And what you have then is faster startup time, you eliminate all of that contracting administrative burdens. All of the data quality cleaning and things like that. And you reduce the time to completion. And this is why it’s important to have a system. And so then you can scale this to many studies and you get much, much lower costs, due to economies of scale. And what one of the things that I’ll harken back to is several years ago, Janet Woodcock made a statement at one of these meetings, where she said, look, we can’t attract top notch epidemiologists if you don’t have a top notch epidemiology program, with a top notch epidemiology system.

We’re getting to that point. And we can attract much better now. The other thing to think about is this is no different than saying to a physician, for the physicians in the audience, look, we’ll take away all your administrative burdens, you just focus on taking care of the patient. What you see back here, in these four boxes is this is what epidemiologists like to do. This is their bread and butter. This is their expertise. It’s not the proceeding four boxes. And so it’s a much more effective system.

All right, and so once you have a system in place, it makes large-scale endeavors far more cost-effective and faster. Any structural improvements that you make capitalize on the existing system. So if we integrate Medicare data, it’s not just integrating a new safety course, it’s integrating it into an ecosystem that has summary tables, L1s, L2s, L3s in place. If you develop signal detection methods, that’s being integrated into an ecosystem that can already to signal investigation and signal evaluation. And so additionally, a unified common platform that enables collaboration with other systems. And this is probably one of the most promising things, and I’m so happy that Robert Platt is coming, is here to talk later in the afternoon. Is when another system adopts the Sentinel Common Data Model, they automatically get the tools.

And now we create a possibility, at least in this case, of a North American system that draws upon the data out of Canada. And for the Canadians, they can help draw upon the data in the US. And so it’s a symbiotic relationship. And there’s a lot of possibility there. Same thing with accessing PCORnet, which is the single largest repository of EHRs. So the fact that they have a very similar system means growing towards them is much faster. The other thing is you can also expand things beyond post-market safety.

And so at FDA, we talk about this FDA-Catalyst, and this is run by the Office of Medical Policy. And we’ve heard already about the mobile app, and we’ve heard already about IMPACT-AF, but the two things about this is that you have the claims data, you have the ability to now collect patient-generated data. And now you have the ability to randomize. And you start to create the conditions that are ripe for doing distributed pragmatic trials. We’re not there yet, but we’re almost there. All right, so this is my second-to-last slide. It also makes smaller scale endeavors much more likely, much more frequent. It allows us to address a wide variety of agency needs, without the need to martial organizational support every time. For example, medication errors, risk mitigation studies, biosimilars, genetic drugs, our mandate is vast. And I’ll just pick medication errors, where had Jill Wyath, who was leading a methotrexate analysis, had she had to write a five-page proposal, get it approved by her supervisor, get it approved by her supervisor’s supervisor, take it to a committee of other offices that then she has to defend.

And then if her gets approval there, if it doesn’t align with the budget timeline, then she has to wait a year for next year’s budget. All of that, and that’s not even talking about the contracting yet. Several years later, that’s when her idea becomes reality. And here, she was able to walk into our office and say, look, we wanna try this out, can we turn it around in six months? So it makes for a much more innovative system, it makes for a much more positive work environment for FDA.

So in summary, Sentinel is a mature semi-automated safety surveillance system that is distinctive in its use of parametrized, reusable analytic tools, combined with a continuously quality-checked multisite database. It provides a single common platform to help meet FDA’s needs in evaluating the safety of medical products after approval. And the challenge for FDA, and Sentinel, and its collaborators, is to build upon these successes to date by continuing to advance the core deliverable, which is post-market safety, but at the same time, expanding into all these other areas of need. Thanks a lot. – Great, thanks. Thanks, Michael, so we’ll turn it over Azadeh.

– Good morning, my name is Azadeh Shoaibi. I’m the lead of the Sentinel program at the Center for Biologics Evaluation and Research. And I’m going to, I’m going to present a key achievements of the CBER Sentinel program in 2017. The products that are regulated by CBER are diverse and they are called biologics. We have vaccines, blood components, and blood products, human tissues and cellular products, gene therapies, and xenotransplantation products. So the activities under Sentinel System, or Sentinel Initiative basically look at the post-market safety of this diverse group of products. In the past year, as you have heard from others, CBER has focused on a major priority which was presented in last year’s Sentinel workshop, and that is building a semi-automated national biovigilance program.

We have made some progress in this area, and I will present these achievements later in my presentation. In addition to this new area of development, CBER has continued to integrate Sentinel activities to generate empirical evidence to support safety and also a regulatory decision-making for biologics. We have also prioritized efforts to improve transparency and governance for the CBER Sentinel program. These include establishment of the CBER Sentinel Advisory Committee, which is composed of members from all of our product offices, as well as Office of Biostatics and Epidemiology. And in addition to the engagement, in addition to this CBER Sentinel Advisory Committee, we have also engaged the product offices and set Sentinel priorities for the product offices, so that our work with them will be a more fluent and there will be more engagement on their part, and we have also engaged them in different Sentinel trainings that have been conducted throughout the year. Last year, we also, in the same workshop here, we announced another priority for the CBER Sentinel program. And that is, as you heard from others, to make the program better, faster, cheaper.

And some of the areas of improvement that we thought would be important to take on to make this goal come to fruition is increasing capacity, decreasing costs, reducing the data lag, and also accessing electronic health record data sources. And throughout my presentation, I will provide examples, in terms of activities that CBER undertook in the past year to realize this priority. As you are aware, Harvard Pilgrim Health Care Institute is one. this is moving on its own. – Can you go back one? – All right, Harvard Pilgrim is one of the contractors and our partner for the CBER Sentinel program. So now I would like to present some of the achievements that Harvard Pilgrim group. I’m not in control. That Harvard Pilgrim group has helped us to achieve. So we have increased the use of query tools by about 63% in the previous year. We have also had, so the use of query tools would help us to reduce the time and the cost that is associated with some of the protocol-based activities.

So whenever possible, we have tried to use the tools in lieu of the protocol-based activities. However, we have still had a number of ongoing protocol-based activities last year. And 10 of them were completed. HP group also conducted two training sessions for the CBER staff with regards to the Sentinel program. Here is a breakdown of the types of queries that we conducted last year. We had a total of 41, 37 of them were descriptive, and four of them were analytic. Now these, the results of these queries are used to inform our regulatory decision-makings, and to address safety issues related to biologics. For example, we looked at the geographic incidence of babesiosis, which is a vector-borne disease. And we used that analysis to inform policies related to blood donation and blood testing. We also looked at adverse events related to transfusion of leukoreduced blood and a rate of transfusion in high-risk populations. And we used the results for risk assessment and blood safety purposes. I mentioned that 10 protocol-based activities were completed, most of these activities focused on building the infrastructure and the developing methods, such as safety of vaccines in pregnancy, or vaccine effectiveness feasibility.

But we also had a few projects that looked at the potential association between specific medical products and specific adverse events. Biologics, as you may have heard previously, have certain characteristics. And because of these characteristics, the surveillance program for biologics also requires certain capabilities. So in order to expand and enhance the CBER Sentinel program, and to also customize the components of this program, for the safety surveillance of biologics, in 2017, CBER awarded two new one-year contracts to IQVIA Institute and OHDSI working together. As Steve Anderson mentioned, the name of this new program is Biologics Effectiveness and Safety initiative.

IQVIA Institute is formerly known as QuintilesIMS, and it’s a private company. OHDSI stands for Observational Health Data Sciences and Informatics, and it consists of a network of interdisciplinary scientists with an open source platform. The slides are moving faster than I am. So is that a sign? (audience chuckling) So the two contracts that we awarded last year. The objective of what we call contract one is to develop additional surveillance capabilities as specifically required for biologics. And access to EHR data sources is a major component of this requirement. And the objective of what we call contract two is to utilize innovative methods, such as machine learning, natural language processing, and artificial intelligence to mine unstructured data such as physicians notes or extensive pathology results, et cetera, in EHR data sources. So here, I’m showing the current model of the CBER Sentinel program.

We have two major contracts. One is with Harvard Pilgrim Health Care Institute, and the other one with IQVIA-OHDSI collaborative under the BEST initiative. So now I would like to focus on the achievements of the BEST initiative in the last three months of 2017. This is moving. – They’re trying to fix it in the back, but if we could. – Okay, so it’s very distracting to me. So I was going to say that we, I’m going to talk about the achievements of the BEST initiative in the last three months of 2017, after the two contracts were awarded. So the BEST initiative has access to EHR data sources as well as claims data. IQVIA provides about 160 million patient records from claims sources, 83 million patient records from billing data of hospitals, and 44 million patient records in the form of EHR from ambulatory care settings. We have other data partners, including Columbia University, Stanford University, University of Indiana, and Regenstrief Institute. And these data partners provide about 24 million EHR patient records.

And all of these data partners transform their data into the OMOP Common Data Model. And they refresh their data quarterly, so we have access to data that are about three months old. The BEST initiative uses the OHDSI tools. And these tools are freely available online for FDA’s staff or any other scientist to use them. And the tools do a range of activities, including generating protocols, validating R packages, they generate the validated R package, figures, tables, and also draft manuscripts. So here, I’m showing a screenshot of one of the tools called ATLAS. ATLAS does a number of activities, such as defining cohorts, generating patient profiles, conducting effect estimations, and other activities.

So because these tools are freely available to FDA’s staff, they can easily, with some spending a short amount of time at their desk, design their own queries or epidemiologic studies. Here, I’m showing an automatically generated protocol by the ATLAS tool. When you specify the parameters that are needed for a query or a study, then your protocol is automatically generated. So this automation ensures that all parameters are documented, it improves transparency for the exact specifications executed, and it also eases reproducibility.

Similarly, one can select the R code option to look at the validated R package that is generated after one specifies the parameters for a study or a query, and the R code can be modified manually if needed. The tool also generates figures and tables. We can use this feature for characterizing the available databases or a specific cohort. Some of the specific technical work that we have done within the BEST initiative in the last three months of 2017 include building a library consisting of multiple coding systems for blood exposure identification in both claims and EHR databases.

This library has about 4,000 codes and we have queried all of these codes within all of our data partners. We have also incorporated ISBT-128 coding system into the OMOP Common Data Model. And that coding system has about 14,000 codes for blood and other products. Now ISBT stands for Information Standard for Blood and Transplant, and it is a global standard for identification, labeling, and information transfer of medical products of human origin. Our experience with claims data has shown that claims data do not have sufficient amount of data and granular enough data for surveillance of blood components, blood products, tissues, and other advanced therapies. So with the availability of ISBT-238 coding system, we have seen and studied this coding system and we have realized that this coding system, which is a barcoding system, using blood banks and hospitals, has very granular data for our products.

And that’s one of the reasons that we incorporated ISBT into the OMOP Common Data Model as one of the first steps we took. And also, the BEST initiative conducted three training sessions in the last three months of 2017. So the main accomplishments of the BEST initiative is the expansion and enhancement of the CBER Sentinel program, with respect to data and infrastructure so far. So we have added EHR data sources.

We are using a flexible and expandable common data model. We have reduced the data lag significantly. And we have access to query tools that are available to everyone. And also, we are using innovative methods to identify and report blood transfusion related adverse events. And this program has also created a significant cost reduction. So we awarded two new contracts last year to the BEST initiative. But both of them are, both of them are one-year contracts.

So in order to continue the activities of the BEST initiative, CBER initiated a market research for two potential new BEST contracts in fiscal year 2018. And here, I’m showing the, it’s moving. (chuckles) – Thinking that perhaps it’s double-clicking, it might be sensitive, so maybe just – Okay. (muffled speech) So here, I’m showing the website address for the request for information. The two requests for information that were released in January of 2018. In conjunction with these two RFIs that have been released, CBER is hosting an industry day. It’s a public meeting, we call it industry day, on February 12th, which is this coming Monday at the White Oak Campus. And this is the registration site for that. And the purpose of the industry day is to provide an opportunity for potential vendors to ask questions about these two RFIs that have been released.

So in summary, CBER has made significant improvements to the biologic surveillance system, by addition of EHR data sources, advanced tools, reducing the data lag, and using a flexible and expandable common data model. We continue to integrate surveillance activities of Sentinel into the regulatory decision-making. And we have, by managing our existing resources, we have made the surveillance activities and the program more sustainable, cost-effective, and efficient. And these are steps taken towards building the semi-automated national biovigilance system that I mentioned earlier. And I would like to acknowledge a large group of people and many entities who contribute to ongoing work of Sentinel activities, including more importantly, my team, CBER Sentinel Central Team.

– Okay, great, thank you very much. Apologize for the technical difficulties. We are running quite a bit over on this panel, so I’ll quickly turn to Brian from Amgen, to get some of your reactions, thank you. – Okay, so I’ll start by saying I’m glad I don’t have slides. So thank you to the organizers for the opportunity to be here today, to Drs. Nguyen, Shoaibi, and Kashoki. It’s been very helpful to talk with you in advance of the meeting and to hear your comments here today about Sentinel’s achievements, and the new initiatives that the agency is embarking upon. The FDA Sentinel System represents a significant advancement in our ability to systematically conduct post-marketing drug safety research and analyses. As a taxpayer, a husband, and a father, I personally believe that having the ability to rigorously monitor the safety of medicines before and after they receive regulatory approval is critical. And as a pharmacoepidemiologist, I believe that the use of healthcare databases combined with an appropriate study design and robust analytic methods can yield important and reliable information about the use and safety of medicines in many circumstances.

So I work in the Center for Observational Research at Amgen, and I lead our data and analytics center. We sit in the development organization and are made up of epidemiologists, biostatisticians, statistical programmers, and data scientists. Our group is charged with developing and advancing the use of real-world evidence to support medicines throughout the drug development lifecycle. An important element of what we do is to provide robust epidemiologic evidence regarding the effectiveness and safety of our medicines. Nearly three years ago, our team downloaded all of the Sentinel modular programs into our real-world data platform, which has been harmonized to the OMOP Common Data Model. We built an ETL process to convert data from the OMOP model to the Mini-Sentinel model so that we could run Sentinel modular programs without affecting their underlying integrity. We decided to integrate the modular programs, so that we’d become more familiar with and better understand what they were doing, but also the leverage the extensive methodologic work that had been incorporated into their development.

Today, we use these programs to address a wide array of questions, relating to our medicines, both in development and in the post-marketing setting. In the process of doing so, we have learned a great deal and have a better understanding of the strengths and potential limitations of this system, and have taken the opportunity to make adjustments so that these programs could be better address the biologic medicines which we produce. Some of the comments I will give today will reflect some of that learning. So since being invited to participate in this meeting, I’ve taken the opportunity to interview colleagues and stakeholders across the healthcare ecosystem. Epidemiologists in safety organizations, academic pharmacoepidemiologists, government employees, all with the goal of gathering feedback, so that my comments here today reflect a more representative view than simply my own. I’ve synthesized the feedback that I have received into four major domains.

Reflections on the foundational work to build Sentinel, a need for greater education about Sentinel processes, a desire for greater awareness and potential engagement when queries are executed, and opportunities for stakeholders, including industry, to collaborate with the agency in the evolution of the Sentinel System. So first, all stakeholders that I spoke with recognize that the development of the Sentinel System has been a significant achievement. The FDA took the mandate in FDAAA, and designed, built, and operationalized a post-market safety surveillance system that can be reliably used to address post-marketing safety questions. Throughout the process, the agency engaged thought leaders in pharmacoepidemiologic research, and built a system based on sound methodologic principles. And engaged data partners across the United States, who have expert understanding of their data systems. Now that system has over 180 million lives covered, which is quite a sample size, even though it does reduce in certain circumstances.

The system continues to evolve and expand, with a focus on improving data quality, not just getting more data. There is also increasing awareness that the system is being used to answer a wide range of questions, as evidenced by the many ARIA reports that have been released on the Sentinel website. And now, by being integrated into regulatory decision-making, as Dr. Nguyen showed you earlier, as well as label changes, an important advancement. Second, there’s an opportunity to increase awareness about the extensive processes that go into executing safety analyses within the system. It may help to educate stakeholders about the time it takes to develop analytic specifications to ensure that they are done to meet the needs of the study, to work with the data partners in all that’s entailed in doing so, and to execute these analyses and synthesize results.

Sponsors who will work with IMEDS will learn this, but there may be opportunities to broaden awareness, about what is required to do these studies. Those of us who do it know how it complicated it is. But I think more broadly, it is important to federate that knowledge. There is a desire, thirdly, to have greater awareness by industry when queries focused on a sponsor’s medicines are being executed. And if possible, find ways for industry to provide input into how these analyses are conducted.

It may not be permissible within the legislation, but given the significant expertise that sits within the walls of biopharmaceutical companies about our medicines, are there ways that we can play a more active role in understanding the safety of medicines. And finally, if we believe that the Sentinel System should be a learning system, is there an opportunity to engage stakeholders who have worked with the Sentinel System to help engage its suitability for addressing a wide range of safety questions, including those related to biologic medicines. For example, our team, when we integrated the programs, we realized very early on that it was not well-designed for biologic medicines. So we had to tailor it to meet those medicines’ needs. I think those learnings can be helpful to the agency. But additionally, we know in the methods communities that there are a lot of challenging issues in doing post-marketing pharmacoepidemiologic research, like informative censoring and treatment discontinuation.

All of which can impact the reliability and validity of these results. Those of us that are challenged by these problems are developing analytic solutions, and we think it can be helpful to engage with the agency, and others, in the evolution of this system. So I thank you for the opportunity to speak today, and hopefully my comments have been helpful. – Okay, great, thank you very much. Mwango, sorry, I’m gonna have to ask you to maybe err on the shorter side, or the quicker side, if possible, so that we can try to finish. ‘Cause we’re way over. – I totally understand. – Thank you. – I’ve been trying to edit as we’ve been going along. So hello is gonna be my first opening. And I think that just want to say that my comments summarize an OND experience, but the Office of New Drugs, OND, is fairly large and diverse.

So there are unique experiences across the different review divisions, as well as commonalities in the experience. But I’m just giving you the high-level summary of these. And so in terms of integration of Sentinel into the drug review process, and into the specific processes for the Office of New Drugs, as would be expected with any new scientific resource that’s intended to inform regulatory decisions, it has been a period of learning and adoption of Sentinel and ARIA by the review staff within that Office of New Drugs. As Sentinel’s capabilities have matured, so have our staff’s understanding and use of these capabilities. And to the point now, as hopefully you got from Michael’s slides, that Sentinel is now considered an important source for safety information about approved and marketed drugs. To date, within the Center for Drugs, or CDER, the primary users of the Sentinel System are the staff in the Office of Surveillance and Epidemiology, or OSE, and the Office of New Drugs, with a lot of very necessary and helpful input from the biostatisticians in the Office of Biometrics.

The OND and OSE staff collaborate very closely on safety issues, both in the pre and post-market spaces. And engage in a lot of discussions about the potential use of ARIA and ARIA queries, as well as the sufficiency of ARIA when we’re making those very specific determinations as to whether we should require post-market safety study or trial of a sponsor to further evaluate an issue. Through these collaborations, we’ve developed and are continuing to refine our processes for staff discussions with the intent of ensuring that they happen in a systematic way. And particularly in the context of review of new applications, that they happen in a very timely way so that we can meet our deadlines.

Additionally, as we’ve gone through our learning, we’ve summarized the specific considerations that FDA makes when we’re determining ARIA sufficiency to evaluate a safety issue, and therefore, whether a post-marketing safety study or trial is needed. We have enough information to put this into documentation, and so it’s our intent to reflect these specific considerations in either a map or a guidance that can be made available and therefore, more transparent in the specific things that we think about. Michael outlined the really broad framework for sponsor notification, about planned ARIA analyses. And as we’ve committed to in our PDUFA VI letter, we will continue to refine those parameters and make, become more detailed in identifying when we would notify companies about planned ARIA analyses, the extent of information we would provide. And how we would do so. But for now, companies can expect to receive, during the, if they have submitted a marketing application, and FDA’s intending to do an ARIA analysis for a new drug, we would notify them through the approval letter that is issued by the Office of New Drugs.

And in the post-approval space, companies should expect to receive a formal communication form of a standard letter issued from the Office of Surveillance and Epidemiology. It’s been alluded to earlier this morning that a happy consequence of all of this progress that’s been made is that we have even more regulatory questions to consider now. People have mentioned earlier about considerations about whether we can use, or should use, Sentinel for data mining or signal detection. Other things that we have to consider is, we make decisions based on the best available information, and so we have to consider where ARIA analyses and results fit in, in terms of a hierarchy of considering the information that we do wave.

And then also, we sometimes conducted an ARIA analyses when we’ve had either uncertain information or contradictory information, and so then what do we do if the last bit of information that we received is that from ARIA? Recognizing that in and of itself, the claims data and the analyses that we perform do have their own set of limitations. So to sum up, I think we know that observational data has been, and observational studies, have been very useful to us in terms of evaluating drug safety issues. Information that we’re getting now from Sentinel is another form of observational data. And it’s really just been tremendous to have this advanced, and adding to our observational data and study and armamentarium. I hope I made sense, I talked really fast. Thank you. – Great, thank you, I appreciate you taking all of the presenters and going through this rather quickly.

So I’m gonna use my prerogative as the moderator to allow us to go a few minutes over. It’s really important that we have the opportunity for the audience, we might have time for one quick question. First one to the mic. While they’re doing that, I do have one question that came up a couple of times, IMEDS. We are gonna hear about IMEDS later in the afternoon, but from a user’s perspective, not from the FDA’s, and if you don’t know, IMEDS is a program offered by the Reagan-Udall Foundation to enable the industry to have access, and to engage with Sentinel System partners and tools. Michael, you mentioned ARIA and how it’s becoming an integral part of FDA processes and decisions, even the approval letter that included a planned post-market analysis using the ARIA system. Brian, you mentioned that stakeholders would really like to collaborate better with the system, and mentioned IMEDS as well.

So my question to Michael, or Mwango, is, when you’re thinking about sort of ARIA in the post-market system, as part of an approval, is that for companies to perhaps engage with IMEDS to do that, or is that more of a planned analysis that FDA would be doing? So with IMEDS available for companies to do, how are you seeing whether or not FDA’s gonna be using the ARIA System, or if the company’s going to be doing that under a PMR? – So thanks for the question. So ARIA, as I said before, is the tools and the electronic data in the Common Data Model.

If we made a decision or a determination that ARIA is not sufficient to study the safety concern, we will issue a PMR, generally speaking, for it. We’ve created IMEDS to open up Sentinel to industries so that they could satisfy the PMR in Sentinel, if they wanted to, and through that interaction, then you use the normal bilateral relationship between FDA and industry in the context of a PMR to collaborate on that PMR. But in general, we don’t specify a system ever. But certainly it’s a system that is available and that we’ve accepted as a source. – Great, thank you. I’m gonna, so nobody, so I’ll take one more. And this is to Azadeh about the BEST system. So we heard quite a bit about the system this morning, there’s sort of the rationale and the needs for blood products and the like, in terms of richer data to sort of be able to measure the important exposures and outcomes. And you certainly talked about it in your presentation. But for those who, in the audience, who may not be fully aware of the differences in the system, BEST, you showed a lot of data availability, including a lot of electronic health records.

And the Sentinel system that Harvard Pilgrim is operating, also has data partners that have electronic health records. Is there overlap or is there a plan to, for some analyses to dip into sort of both systems, because there might be overlap in patients and in data types? – Well, I think the Harvard Pilgrim component of the Sentinel program, CBER Sentinel program and the BEST initiative component, each one has its own place.

With respect to data specific to one group versus another, I think if I’m not mistaken, about perhaps 10% of the data providers for the Harvard group have electronic health records. Versus the BEST initiative, which perhaps provides us with more EHR data sources. But I think the issue is more related to how that data is actually utilized and that would be related to the common data model. Whether a data model is expandable and flexible enough to be able to quickly adopt these existing and new coming coding systems, such as the ISBT coding system, or not, that would be an issue that would make a difference for us.

Otherwise, we are not, we are kind of agnostic to what common data models should be used, as long as it can provide us with the data that we need. And also, the data lag is another issue that we need to have access to data that are more recent, than for example, nine to 12 month lag that we get from the Harvard system. And the other issue is related to the tools that we have, for example, HCA EHR data available, but our tools are not, the Harvard Pilgrim tools are not ready yet to run queries on the EHR data that HCA provides for us. So I think each system has its own place and advantages and disadvantages. But these are some of the issues that we have experienced with respect to blood. – Great, thank you. I’d like to thank all of our panelists for such great presentations and comments.

I’ve got good news and bad news. The bad news, I’ll start with that, is I’m taking five minutes away from your lunch. So we are gonna start back here exactly at 1:15. But the good news is you can bring your food back here, if you’re running out of time. And you certainly bring it in here. There are restaurants in the area, and Sarah at the registration desk can give you a sheet that has some of those listed. Thank you. (audience clapping) – All right, thank you. – How are you? (people chatting) – Set of panelists to join me up here on the stage. Hi, how are you? Good to meet you. (people chatting) Should I get started? Yeah, okay, evolution of the Sentinel System. In this next session, we’ll discuss some examples of how access to and use of the Sentinel System’s tools, data, and infrastructure is expanding. And in increasingly more diverse ways, Sentinel is being used to generate a range of evidence. We heard a little bit about that this morning, this next session includes a set of panelists who all have various experiences in engaging with the Sentinel System, or in the case of CNODES, complementary systems in Canada.

Joining me on the stage are Allison O’Neill, Epidemiologist at the Center for Radiologic Health at FDA. Chris Granger, Professor of Medicine and Director of the Cardiac Intensive Care Unit at Duke University Medical Center. Robert Platt, Professor, Departments of Pediatrics and of Epidemiology, Biostatistics, and Occupational Health at McGill University, and Co-Principal Investigator of the Canadian Network for Observational Drug Effects Studies, CNODES. Beth Nordstrom, Senior Research Scientist and Executive Director of Epidemiology at Evidera. And finally, Claudia Salinas, Senior Research Scientist, Global Patient Safety at Eli Lilly. And I think I’ll go ahead and turn things over to Allison. You’re welcome to. – Is there a clicker? – Yeah, I’ve got it right here.

– Oh, thank you, good afternoon. My name is Allison O’Neill, and I’m an epidemiologist in the Center for, whoops, there we go. Devices and Radiological Health at FDA. And I have been asked to present today to describe the use of the Sentinel infrastructure for the evaluation of a medical device. I’ll also touch on key challenges with monitoring safety of devices versus drugs, as well as opportunities to continue leveraging Sentinel for medical device surveillance. We performed a device evaluation of use of laparoscopic power morcellators, which I’ll refer to as LPM.

So just a bit of background, uterine fibroids can be removed via laparoscopic procedures, which are also referred to as minimally invasive surgery, which are performed with or without LPM. Benefits of minimally invasive surgery may include shorter hospital stay, faster recovery, reduced wound infection, and reduced incisional hernias, compared to using a more traditional larger abdominal incision known as a laparotomy. LPMs are a class II device, which are inserted into the body and used to cut fibroid tissue into smaller pieces for easier removal. However, a safety concern has been identified with LPM, which is that uncontained intraperitoneal morcellation of uterine tissue which contains an unsuspected malignancy can result in dissemination and upstaging of that malignancy. The FDA estimates that between one in 225 to one in 580 women undergoing surgery for fibroids may have an occult uterine sarcoma. So an immediately in effect guidance document was issued by FDA on April 14, 2014, to communicate new contraindications and a black boxed warning for LPM during laparoscopic surgery.

And after FDA issued the guidance document, there was anecdotal evidence that the availability and use of more morcellators decreased drastically, due to actions by some manufacturers, payers, and hospitals. Therefore, some asserted that use of minimally invasive procedures decreased as well. This is a table published online by FDA in December that shows seven publications reporting on the changes of percentages of various procedures between 2013 and 2015. It generally suggests that the percentage of women undergoing open abdominal procedures was higher than prior to 2014, although reported changes varied between reports. Two reports showed a large decrease in the use of LPMs, but there were mixed results for any change in percentage of laparoscopic and minimally invasive surgical techniques, although they remained a common procedure in all reports. So we decided to analyze the Sentinel data to assess the rate of LPM use and laparoscopic surgeries before and after FDA’s 2014 actions. So we wanted to look at the following study questions. Did use of LPM decrease after our safety communication in 2014? And also, did the proportion of procedures performed laparoscopically also decrease in favor of abdominal or vaginal procedures? Here’s just a quick look at our methods.

The date range was 2008 to 2016. All women with six month continuous enrollment and no previous hysterectomy were considered eligible. We then identified women receiving hysterectomies or myomectomies to remove fibroids, except for radical procedures to remove cancer. And we grouped them into laparoscopic, abdominal, and vaginal procedures. We then used a HCPCS Level II code to identify procedures that used a morcellator. For our results, this slide shows the use of the morcellator code over time. The vertical purple line shows the month of April 2014, the date of the safety communication. And the use of morcellators, as identified by this code, had been increasing from 2008 to 2014.

And was then seen to decrease for both laparoscopic hysterectomy, shown by the red line, and laparoscopic myomectomy, shown by the light blue line. After April 2014, there was no evidence that uterine fibroid surgeries decreased during this time. And this one shows the percentage of procedures which were performed laparoscopically with or without morcellation. The percentage of hysterectomy procedures performed laparoscopically shown on the left was rising from 2008 to 2012, when it plateaued around 60%. It declined slightly around 2014, but stayed around 60%. The percentage of myomectomy procedures shown on the right was rising from 2008 to 2013, when it plateaued around 35 to 40%. After 2014, it decreased below 35%, but it increased again to its original peak. So therefore, it is possible that the decreased use of LPMs after the safety communication may have been associated with a plateau or temporary decrease in the laparoscopic approach for these procedures, but it did continue to be used at a high rate. So to summarize, we observed that the use of LPMs, as reported by this code in hysterectomy and myomectomy procedures drastically declined after FDA’s safety communication. The percentage of procedures performed laparoscopically experienced a small decline and a plateau, but a large decrease was not seen.

This indicates the patients are still receiving laparoscopic procedures without morcellation, which have benefits for patients recovering after surgery, such as those discussed earlier. In 2017, FDA conducted an updated assessments of LPMs, including in this analysis as well as review of literature and medical device reports, or MDRs. Available evidence was consistent with our previously estimated prevalence of occult sarcoma, and worse patient outcomes for morcellated malignancy. FDA continues to warn against using LPM in gynecologic surgeries to treat patients with suspected or confirmed cancer, and in the majority of women undergoing myomectomy or hysterectomy for uterine fibroids.

The Sentinel data was valuable to the team in an attempt to assess the impact of our safety communication on clinical practice. The analysis confirmed that the use of morcellators declined as recommended by the communication across multiple healthcare systems. So the main challenge for this analysis was related to identifying the procedures using LPM with a CPT or ICD code, which is a challenge that is relevant for any source of claims data. As we heard this morning, regarding some unique data needs for medical devices, devices do not have a National Drug Code and we must rely on a CPT or ICD 9 or 10 procedural code used for billing. The procedural code may or may not capture the use of one or more devices.

And in many instances, the brand of device is not known, unless there’s only one brand on the market at that time. In this case, we used a Level 2 HCPCS code, which is simply labeled morcellator, and it does not provide detail regarding the type or brand. It was also difficult to assess the consistency with which that code is or was used in practice. And finally, we kept this simple with a Level 1 analysis of background rates for these procedures to get a snapshot of how the rates changed over time without more advanced time series analyses. So there have been challenges associated with the use of Sentinel data to assess use of medical devices, due to some of the obstacles and limitations discussed. However, there are definitely opportunities to utilize Sentinel for more analyses for devices that can be identified using procedural codes. And in the future, as the Unique Device Identifier, or UDI, continues to be adopted and incorporated into EHR and claims data, we’ll be even better able to identify use of the specific device of interest. Additionally, CDRH is in the process of building NEST, which is our National Evaluation System for health Technology, which will be a collaborative national system to link and synthesize data from different sources across the medical device landscape to generate evidence across the total product lifecycle.

As you heard from Greg Pappas this morning, the development of coordinated registry networks within NEST provides the opportunity for potential collaborations and data linkages. As part of our long-term goal to expand our ability to assess real-world safety and effectiveness of medical devices. So in conclusion, this was a valuable resource to the CDRH team to assess background rates of device use and possible impact of FDA’s actions on real-world use and practice. And it provides a demonstration of how the Sentinel infrastructure may also be used for medical devices. As CDRH moves forward in the development of NEST, using UDI and device registry networks, and as the Sentinel System also matures and evolves, for example with the use of EHR, there will be great opportunity to perform more detailed and accurate assessment of device performance in the real world. Thank you. – Great, thanks, Allison. We’ll turn to Chris. – Great, thanks.

And I am delighted to talk for a few minutes about the IMPACT-AFib trial, which you’ve already heard about from Rich Platt and others this morning. But this has definitely been one of the most exciting and innovative things that I’ve ever been involved with. And I will share some thoughts about it, but begin by noting that there was a Collaboratory Grand Rounds on January 5, that went into the project in some detail with the link here for those who are more interested in hearing more complete details about the program. I’ll also just mention that this morning reminded me that there is this fine line, as we talk about evidence generation and safety. And that, in fact, in this case, which I’ll make the case that here we have an example where we have a treatment that prevents almost all strokes for a common condition that’s only used in about half of patients for whom guidelines indicate that it should be used.

And I think that’s also a safety issue. That when we have a clearly effective treatment that’s not being used, that that’s a public health safety issue as well as efficacy issue. The genesis of this program came in part from a partnership with Sentinel and the Clinical Trials Transformation Initiative back in 2014, with a plan to use Sentinel to explore conduct of randomized clinical trials. And so IMPACT-AFib then is the first of such trial, with this direct mailer to health plan members who are at high risk for stroke and not on oral anticoagulants. To encourage these members to go and discuss with their provider whether or not they might be on an anticoagulant. Rich has shown you this here, the five data partners on the left, and the research partnership on the right, including the patient representative, and the Clinical Trials Transformation Initiative.

Atrial fibrillation is the common arrhythmia. It’s more common in older people, it occurs in, about five million Americans have atrial fibrillation. It causes about one in five strokes, or about 20% of all strokes are caused by atrial fibrillation. And we can prevent about 70% of those strokes with oral anticoagulation. So we have an extremely effective therapy for preventing those strokes. But what we see in a variety of data sets, but most relevant from the Sentinel data itself, that only about half of patients have been, in the data set, who have atrial fibrillation, risk factors for stroke, categorized by this commonly used clinical risk score, the CHADS-VASc score, have been treated with anticoagulants, and even fewer are currently being treated with anticoagulants.

We did a cluster randomized trial published last year to show that a largely educational intervention can improve the proportion of patients treated with anticoagulants who have indications for that treatment. And that that treatment was also, that that intervention was also associated with a significant reduction in stroke. So the study design then for the IMPACT-AFib trial is to take patients who have atrial fibrillation based on at least two claims, one in the prior year, CHADS-VASc of at least two, no recent admission for bleeding, and not on any anticoagulant for the past year. Defined as no claim for a prescription or INRs that have been obtained which could identify patients who are on low-cost Warfarin, where they might have paid with cash and not entered a claim.

And then the plan is to randomize 80,000 patients. Now Rich tells me that as of today, we’ve got 38,000 of the intervention mailings have gone out. So we’ve got a total of 76,000 patients now randomized and anticipate the remainder of approximately 80,000 to be randomized, and with the intervention implemented in the next couple of weeks. The aim is to increase the use of oral anticoagulation, defined as at least one prescription filled, but we’ll look at a variety of other outcomes.

Iincluding clinical outcomes, admissions for stroke and bleeding, and the number of patients who remain on an oral anticoagulant based on claims at the end of one year. The intervention, Rich has shown this, primarily focused on patients, with a carefully constructed packet from the FDA, Harvard, Duke, and the data partners that outlines the rationale of why it’s so important for patients on atrial fibrillation to be on an anticoagulant to prevent stroke, including a pocket card designed to facilitate the interface between the patient and the patient’s healthcare provider. With the intent of using the patient as an agent of change, to help prompt the provider to treat the patient with an oral anticoagulant if that is appropriate. The providers also are being notified, but we actually don’t expect that to be a major part of the intervention. And then after one year, even in the control group, the providers will also be notified. And this is the series of mailings that have occurred, again, now up to 38,000. And of the patients identified with atrial fibrillation, 36% of them are the total of, about 40,000 patients were eligible for the study. Importantly, we also did not require, or did not obtain informed consent, because of fulfilling criteria for waiver of consent.

So we can truly have an unselected population and a truly pragmatic trial to test the intervention. I’d like to acknowledge the team of people who have worked incredibly hard on this trial, that are shown on this slide. And I’ll conclude then with the thoughts that the IMPACT-AFib team has shown that it is possible to conduct the sufficient large pragmatic randomized clinical trial using the Sentinel platform, and addressing a critical health issue. And now that we know that it can be done, it’s important to explore other opportunities to do further randomized trials with the Sentinel infrastructure. And the types of interventions that are appropriate at this point include low-risk simple interventions, like patient or provider education, that are addressing large gaps in care with large potential to improve important clinical outcomes.

And I think there are lots of examples of that. Ideally, or at least something that I think is particularly important in the IMPACT-AFib trial, avoiding the need for patient consent, in order to get a truly unselected population. And some of the possible clinical gaps to address might include statin intolerance. So here we have, especially for patients with established vascular disease, a life-saving therapy that prevents myocardial infarction, stroke, and death, with about 20 to 40% of patients having perceived intolerance, but knowing that only about 5% of the time is that true intolerance. So lots of patients not being treated who could be. Another example is diabetes with vascular disease, where we know that only about 14% of those patients are treated with what the American Diabetes Association deems is, or the treatments which have an evidence base for use. Thus, a major opportunity also for improving care there. And there are lots of other opportunities. I think there’s a long list. And one of the keys, I think, is to identify the most important opportunities, to identify funding sources, including the NIH, and then do feasibility studies to see what might be most appropriate to address in the Sentinel infrastructure.

Thanks. – Great, thank you very much, Chris, Robert. – Thank you, perfect, okay. So I’m gonna be talking about a little bit about the Canadian Network for Observation Drug Effect Studies, which a few of the speakers described a little bit this morning as the Canadian Sentinel. And some of the work that we’re doing now together with Sentinel on some collaborations. I have a couple of boilerplate slides that I’ll skip over. The bottom line is CNODES is a national network with sites across Canada. My colleague, Samy Suissa is the Nominated Principal Investigator. The coordinating center, and the bulk of the administrative and methodological team is located at McGill in Montreal with us. This is also important to note, it’s funded by our Canadian Institutes of Health Research, which is our NIH equivalent as a contract, as opposed to funded directly through Health Canada.

So our processes at CNODES are very similar to what’s done at Sentinel on a slightly different scale and with some different approaches. We receive primarily safety queries from our stakeholders. And our main stakeholder is Health Canada, similar to here. We’re asked then to, based on the query developer protocol, refine the query, develop a statistical analysis plan, implement a statistical analysis plan, and then do this in a distributed way, similar to Sentinel, where the analyses are done in the individual CNODES sites, brought back to the coordinating center, and then meta-analyzed.

Again, in a process that’s similar to Sentinel, but with a lot more hands-on than what you heard about this morning with the distributed analytic tools that Sentinel uses. This is an example of a CNODES output. Looking at high-potency statins and diabetes, it essentially looks like a standard meta-analysis with the different Canadian sites as studies. You’ll note in there that we include CPRD from the UK and MarketScan from the US as sites as well. And this is primarily to increase sample size, and in the case of the CPRD, and in both cases, to get comparison groups relative to Canada.

Because as Richard said this morning, big data are sometimes not very big, when you actually get to analyzing the numbers, and you’ll see some of the case numbers are quite low, even in some of the bigger sites. Ontario is one of the biggest, is the biggest province, and there’s still relatively few end points there. I’ve shown this slide several times in the last few months, some people may have seen this before. This just illustrates a typical CNODES timeline, and the reason I’m showing this is it takes a long time. And people, the speakers this morning made reference to this as well, it takes a long time to conduct one of these distributed network studies where you’re developing the study, the analysis plan, the analysis, the code, et cetera, on a study-by-study basis.

So this one, as an example, took about two years from the initial query to finalization of the results. Some of the advantages of this approach, I think are fairly obvious. One, we have a lot of analytic flexibility, we can do, we’re not limited by the tools that, by the analytic tools, we can design a study as complex using whichever data sources from each site as we want. We can use statistical methods that may or may not be part of the standard toolkit in a distributed setting.

We’re also doing a lot of capacity building in this process, as I said. We have analytic sites in each of our provincial data holders. And we’re building capacity in data analysis and in methods across the country, as part of this process. And then finally, and this is the case with Sentinel as well, the query author is actively involved in the question refinement and the protocol development, so that we are actually sure that Health Canada’s problems are being addressed, or their questions are being addressed by CNODES studies. So the obvious problems or challenges with the approach that we’ve undertaken up until the past year are, the obvious main challenge is timeliness. Data access is a problem in some provinces, but that’s really sort of a different question. But this efficiency in protocol development, every time we’re developing a new protocol, it takes time, a new analysis plan takes time. If the code is something we’ve done before, if the definitions are things we’ve done before, we can gain some efficiency there, but we’re still essentially designing a new study each time.

And Richard and others made reference to this this morning, that this is, can be a slow process. We try to use standardized definitions and as much as possible, we use standardized and tested SAS code, but that’s in a somewhat informal way, because we haven’t been developing this sort of as a package in the same way that the Sentinel team has done so. So this led us to about a year ago, a little under a year ago now, to decide we wanted to try to implement a common data model and implement some of the similar tools to what Sentinel uses. We were funded for this pilot extra project in April of 2017, and we started with our three sites that have essentially live data access. Where the goal was, well, we should be able to produce query responses in two to three weeks. We implemented more or less the Sentinel Common Data Model tables, enrollment demographics, dispensing, et cetera. And we followed very closely, or are following very closely, the Sentinel Common Data Model structure. And I’ll talk a little bit later about the reasons why we’re doing that. I think it’s important to have that discussion.

And then we’re working through some, we’re planning to work through some demonstration queries with our advisory committee and with Health Canada in the coming months. So far, we’ve developed the table conversions, we have the data in the Common Data Model format. It’s relatively straightforward, it’s a bit time-consuming, just because people have to go through the tables and modify, write code to modify the data structures a bit. We’ve discovered a number of minor decisions that we’ve had to make along the way that have to do with fields or structures that are different in the Canadian sites than in the US. And we’ve prioritized the queries, we know which queries we’re going to attempt when we get to actually have the data online.

Validation of the CNODES Common Data Model is in progress right now. I had hoped that we would be able to present some of the validation work to you today, we’re a little bit behind schedule on this, ’cause for some technical reasons. We’re going to implementing standard Sentinel QC tools, and then reporting back to Sentinel on our validation steps. We’re also going to then run some queries three different ways. We’re gonna use some simple queries, do it using the Common Data Model, using CNODES standard tools, and then using the Canadian Institutes for Health Information, which is another data holder in Canada that can do drug utilization queries quite straightforwardly as a third validation step. Informally, also, we’re working to collaborate with Sentinel. And we’re working to sort of translate and then back-translate some CNODES studies into the Common Data Model and then back into the standard CNODES format. We’ve done it MarketScan, Rich presented some of those results at the ICP meeting this past summer.

And we intend to redo that in a Canadian site as well, just to test the compatibility of the systems. So I’ll speak just briefly on why we chose to work with the Sentinel Common Data Model. There are lots of other approaches we could have taken to standardizing our data. Most of them have been pragmatic, though. We have a close relationship with the Sentinel team.

We’ve worked well together on a relatively informal basis so far. They have a demonstrated process of working with their regulator and that was appealing to our regulatory colleagues to say they knew how the workflow could work, or they could observe how the workflow worked at Sentinel. Our core data tables are insurance administrative claims data sets, so the data sets match very nicely with the ones that they worked with, so it was an easy mapping from one set to the other. And as I said earlier, we’re taking advantage of Sentinel’s well-established data quality assurance processes and procedures so that we know that we’re getting good, we’re getting good data out of the CDM. And finally, and this is perhaps one of the most long-term, but exciting perspectives on this is the potential for cross-jurisdiction collaborations between Canada, and Health Canada, and the US FDA, and potentially, down the line, the EMA, and some of our European colleagues for multi-national studies of drug safety.

Very briefly, there are some technical reasons why we chose the Sentinel Common Data Model as well. I’ll really focus on the first one. Given that the query tools that Sentinel has produced and made available are readily available and match with the Sentinel Common Data Model, and are of proven value to FDA and the regulators in the US, it’s natural that we can use these in Canada. In the future, as I said, we are hoping that this will lead to rapid responses to simple queries from Health Canada. And it should enable these cross-jurisdictional collaborations that I mentioned. And I’ll just focus on this last bullet, you here in the US obviously have a much larger population than we do in Canada, but with our public insurance programs, we have the good fortune that we have a much longer average followup, people tend to stay in one province and stay therefore with the same insurance provider for a long time, so we can look at much longer followup studies on average than may be possible in the US.

Some last concluding thoughts. The Sentinel data structures and the analytic tools have been easy to implement. As I’ve said, and I’ve repeated this, the possibility for Sentinel/CNODES FDA/Health Canada collaborations is seen as a real strength. And this is what we’re really hoping, and our Sentinel colleagues have shown this, that the Common Data Model will advance our response times for some queries. In particular, for things like utilization queries and event rate queries, so that we can get rapid responses to quick questions from Health Canada. We don’t think, and I think we heard this this morning as well, that it will eliminate the need for those complex studies that I showed in the first set of examples, where we use our, we design a study from scratch, but it will help us reserve those for the studies that are really needed. And thank you very much. – Great, thanks, Robert. I’ll pass things down to Beth. – No, there we go. All right, I’d like to start by acknowledging the team, who worked on the study that I’ll be telling you about, including Jason Simeone and Samuel Huse from Evidera, and Kwame Appenteng and Milbhor D’Silva from Astellas.

Now, back in the day, when the Sentinel Initiative was still mini, and they started releasing things like protocols and programs out into the public domain, some researchers realized that they could use these tools, not just to see what was going on in the Sentinel program, but also to do their own studies. So all you had to was take your claims, or possibly even the structured field from an EMR database, get them into the right format, and then run them through your very own of a Sentinel analysis. So that’s what we did. There was a Mini-Sentinel project looking at safety of mirabegron, a treatment for overactive bladder. And the outcomes in that study were MI and stroke. And we set out to replicate that study exactly, but using different data sources, specifically, the MarketScan and PharMetrics large claims databases. Now our approach, unlike something that Rich Platt referred to this morning, this was really quite simple and streamlined. You’ll notice that we didn’t start by writing a protocol, because we were able to simply use the published Mini-Sentinel protocol. So our first step was just to transform the data into the Common Data Model. Then we created lists of the drug codes that were needed, built input configuration files, and ran them through the programing modules that did all of the cohort selection and analysis for us.

Then we did some extra descriptive tables on the side. We did a little bit of reformatting of the results and wrote it all up into a report. It was a very streamlined process really, compared to typical project. Now I won’t walk you through all the detail on this slide, you don’t really need to worry about the inclusion and exclusion criteria, but if you can look at the bottom two boxes, there were actually eight different analyses that we did in this. So there were two databases, there were two outcomes with slightly different patient populations going into each. And then there was a primary and a secondary analysis based on whether the patient had previously used treatments for overactive bladder or not. And within each of these separate analyses, there were tens of thousands of patients included in the analysis.

So you can see that this was by no means a small-scale study. Now the analysis that was done by the module did propensity score modeling and matching. It output all of the descriptive results on the events, the denominators, and it ran Cox models comparing the risk of each event in mirabegron to oxybutynin, which is another OAB treatment. Now the module didn’t automatically produce the baseline pre and post matching descriptive tables that we wanted to see, so there we did just a tidbit of our own custom programing to create those tables. Let me walk you through just some high points of the results. In essence, the MarketScan matching balance worked very well, but PharMetrics, the matching didn’t work at all. The model didn’t converge, and they way that the program was set up was if your model doesn’t converge, you don’t get matched results, end of story.

And there was nothing you could do about it. Now this was a version of the program from a few years ago. And of course, it’s been updated since then. I understand that this is no longer an issue the way it was back then, but you can imagine at the time, this was something of a stumbling block for us. But nevertheless, we went ahead and we ran our outcome models. And we found no evidence of an increase in MI with mirabegron. And similarly, no evidence of an increased risk of stroke with miragbegron compared to oxybutynin. And these findings lined up very nicely with the findings from the Mini-Sentinel study. So we did succeed in replicating their results. Now let’s talk a little about what was good about this process and what was not so good. There’s one huge advantage, and that’s the speed. It really was very quick. You can trim months off the typical study timeline doing something like this.

There’s also the reassurance of knowing this protocol has been very well thought through, it’s been scrutinized by so many people. It’s really unlikely that there’s some fatal design flaw in it that no one’s noticed. Similarly, for the programing, it’s all been thoroughly validated, not likely there’s a bug hiding in there somewhere. Also, there are some ways that you can customize things. I’ll get back to that notion in a minute. First, let’s talk about the limitations a little bit. There’s one major limitation to this. The others are, I’d consider minor. But the adaptability of the programing is the one huge limitation that we do have to recognize before going into using these tools. Any time you have something that’s this automated and fast, you have to sacrifice some adaptability. So even though they’ve been continually expanding on what these tools can do, there’s always gonna be a limit to that. Now other more minor concerns are the fact that, well, for instance, you can’t put together a list of NDC codes in a protocol and call it final, because they just come out too often with new NDC codes.

So some things like that, you just have to go in and look up for yourself on the fly, you can’t just depend on the published protocol for that. Some of the programing, we had a hard time telling exactly what was going on. And the programs took a little longer to run than we thought they should have taken. This was not analyst time, it was just computer processing time, but still again, if we’d be doing our own custom programing, we probably would could have gone in and adjusted things in the programing to make them run more efficiently for our particular situation. And finally, you might need to do a little bit of your own programing, just to get the output that you want to see in the end.

But taken altogether, I would say if your research question aligns neatly with an existing Sentinel protocol, the benefits of using these tools far outweighs the limitations. Now what we did was a simple replication, just running the exact same analysis in different data sets. But there are other ways that you could branch out some from that and not just do the simple replication. You do need to build these configuration files, where you include, among other things, your code lists. So maybe you want to try a slightly different definition of your outcome variable and run that through the programs as a sensitivity analysis. Again, that would be very quick and easy to do using these tools. Similarly, you could define subgroups based on some baseline characteristics of patients. And again, see if your results hold up in these subgroups. You can also veer far more from the original protocol if you choose, and look at completely different outcomes, or change the drugs that you’re looking at. Now if you go that far from the protocol, you have to be careful, because then, in essence, you have to come up with your own protocol.

You can base your protocol off of the existing Sentinel one, but you do need to think carefully about whether the study design and the analytic methods still make sense for your particular research question. But still, if you can piggyback off of the Sentinel protocol and use the programing modules, you can still save yourself a lot of time. And finally, if use of these tools really picks up, it would be great if we can find a way to share our findings with one another, so we can see what one another is doing, as we go this outside the Sentinel System use of these tools.

I, for one, hope that use of these tools does expand, because I think they’re a great resource. And they can be useful well beyond their original purpose. Thank you. – Great, great presentation. Thank you very much, so, Claudia. – Okay, thank you. So I’m going to tell you about some work that Lilly has been conducting with the IMEDS data and the Sentinel team. Lilly launched a query to provide context about the safety profile of patients with rheumatoid arthritis who are receiving treatment with disease-modifying antirheumatic drugs, or DMARDs. And in particular, we interested in estimating the incidence of venous thromboembolism among these patients with RA. We were also interested in other queries looking at the individual events of pulmonary embolism, and deep vein thrombosis.

Results from the query were included as part of Lilly’s resubmission for baricitinib. So we selected the IMEDS data for two reasons. And I’m gonna give you a little bit of background about IMEDS in just a second. But the first was size. Because we were looking at a rare outcome, we needed a large data source. And the second reason was the availability of validated tools for conducting the query. So IMEDS is a public-private partnership that is manged by the Reagan-Udall Foundation, And I know June Wasser is in the audience here, to permit non-FDA researchers to access the Sentinel data and the Sentinel tools. And in this figure, you can see the similarities and the differences between when FDA accesses the surveillance database, essentially, the Sentinel distributed database, and when non-FDA researchers like Lilly access the IMEDS distributed database.

So IMEDS is a subset of the Sentinel database, and we access it through the Reagan-Udall Foundation. From that point on, the process is the same, essentially. Whether it’s FDA or a non-FDA researcher. And in particular, the same tools are used, and you have access to the same team of experts. So the members of the query team were representatives of each of the groups that I had mentioned just before. And in addition, early on the development of our query, we had the opportunity to discuss the query, and the motivation for the query with the data partners. This also allowed the data partners to ask us any questions that they had, and allowed us to confirm our interest in disclosing the results of the query with the broader community of RA researchers and clinicians. This was a win for both of us in confirming that. So our study population was defined as patients who had at least two RA diagnostic codes within the year before baseline. Patients were required to be new users of DMARDs, and they were following until they had an event. They discontinued treatment with the medication that they were on at index time, or they left the database.

And we began with 75 million people represented in the IMEDS data subset of Sentinel, that was contributed by five data partners, and in the largest treatment group, we had about 69,000 treatment episodes. Essentially, 69,000 patients. So I’m going to share the results with you, but I want to tell you a little bit about the process because it was somewhat different. So typically, we would provide a high-level overview of the query that we were interested.

We would contract, and then we would work to refine the query, provide details, and execute it. And in this case, we needed to actually get all of the details, or most of the details, defined ahead of time, so that we could decide the scope of the work. And when I say scope, I’m talking about the modules. We had a descriptive query, since we were just trying to define an absolute measure of the incidence rate. So we used a Level 1 module. And this is important because the module determines the number of scenarios that you can use. The scenarios are essentially the parameters of the query that use for each treatment group. So an example would be patients with RA who are new users of biologic DMARDs and they’re be followed for VTE. So I will say that this step was probably the biggest learning curve, especially for people who had not worked with the Sentinel team before. And we also had to learn about the limitations of the tool. In particular, the ability to use combinations. And I’ll give an example in our case definition. So once we arrived at a final draft of the specifications, Harvard Pilgrim executed a feasibility assessment in the local data, just to look at the numbers of patients and the amount of person time, so we could evaluate if there were any scenarios that we wanted to eliminate, or that wouldn’t be practical.

We chose to be blinded for this, so that the results of the incidence rate wouldn’t affect what we retained or removed. And really, at this point, for us, we handed it off when we had the final specifications, back to Harvard Pilgrim, and a few weeks later, we had our results. Now that makes it sound like very easy, but I will throw in an additional wrinkle, which was that somewhere around the middle of the second step, we needed to accelerate our timeline. And so between handing off this final specifications and getting the results, and I thank you, June, out there, and all of the team members.

We were sending multiple requests to clarify what the timeline was and asking for status updates. There was constant communication and abundant discussion. And it was as a result of the strong partnership of the team that we were successfully able to get our results and incorporate them into our documents. So this slide is simply intended to illustrate that there are three scenarios here. You don’t need to read it, it’s just intended to reveal that there are a substantial amount of detail in every scenario.

Okay, so for our query, we defined VTE, DVT, and PE, based on diagnostic codes. But for VTE, we used a compound definition as well. And this is what I mean about the limitations of the Sentinel tools for using multiple combinations. So this required use of a different tool, the combo tool, and I’m imaging a very large machine, where you feed things in at one end and something magic comes out the other end. But we were limited in the number of combinations that we could use. So we chose this definition, namely that patients who were diagnosed in the emergency rooms, or in hospital settings, would be considered to have an event based on diagnostic codes. But for any diagnoses that occurred in outpatient settings, we also required them to have evidence of anticoagulant dispensing within 31 days of diagnosis. This was done in an effort to balance the positive, predictive value with the maximizing the ability to detect events. In other words, maximizing sensitivity. Okay, so here, this table is showing the results for the two largest treatment groups, which provide the most robust estimates of the incidence rates of VTE. These are patients treated with conventional DMARDs and biologic DMARDs.

And in particular, you will see that patients who were new users of cDMARDs had an incidence rate of per 100 person years. And patients treated with biologics had an incidence rate of events per 100 person years. Now these incidence rates are not adjusted for age, which is a strong risk factor for VTE. Or for other risk factors, which may differ in distribution between these treatment groups. So I will show you the next slide. Oh, so, right, so here, in this slide, I’m showing you the same results as previously. In the grayed rows is the incidence rates for the cDMARD new users and the bDMARD new users, just as previously. But in addition now, you’re also seeing the age-specific incidence rates. And there are two things to notice. One is that age is a strong risk factor for VTE, as I mentioned. This isn’t surprising, this is also true outside of the rheumatoid arthritis population.

And you see that the incidence rates range within the new bDMARD users from to over two events per hundred person years, as you move from the 18 to 49 year old patients to those 65 and older. The same effect is seen for age within the new cDMARD users. The second thing to notice is that when you control for the effect of age, you see that the difference between the new cDMARD users and the new bDMARD users is attenuated to a large degree. For example, if you look at new bDMARD users, in the 50 to 59 year old patients, the incidence rate is per 100 person years. And looking at the same age group in the new cDMARD users, you have the same incidence rate. Now this controls for the effect of age, but there’s certainly the possibility that there are other risk factors whose distribution also differs between these groups, so really, you shouldn’t actually compare directly between groups. But it does show that once you control for age, you reduce some of the difference between the two groups. Okay, wow, this took much longer when I ran through it previously. And so we had some thoughts about our own query, and about the IMEDS in general, for future directions.

We are planning, in the Lilly query, to have an external presentation and a publication. And we are still working with the IMEDS team to validate the case definition. We are also conducting internal investigations using custom programing to look at sensitivity analyses and replicate the Sentinel query. And for IMEDS, we would recommend expanding access and public awareness of the IMEDS access program. In particular, as I have spoken to rheumatologists who are treating these patients, there’s interest in the results that we’ve seen, but they’re not aware of this data source. And then because we were looking at a rare outcome, I would suggest that considering expanding to other Sentinel data partners, and potentially other sources of data, even maybe primary data collection, would be really useful. I know that David Martin was discussion this morning, he’s looking at ways to collect primary data.

And I would hope that that would be incorporated at some point into the IMEDS program. And then we would also recommend to expand to other non-FDA stakeholders. Not only for access to the data, but also for considering adding their data so that it could be evaluated as a whole. Okay, and then, I wanted to thank all of the partners who were involved in this query. In particular, the strong partnership of Judy Maro, Jane Huang, and David Martin, who really contributed to the success of the query. And June Wasser for coordinating our query, and for being very patient with our requests about the status. Thank you. – Okay, great. Thanks to all of the presenters. It’s been a wide-ranging set of examples of how to further the Sentinel infrastructure, how to use it for other things by different users. Great examples, I have a few questions of followup as folks in the audience think about questions that you might have.

Maybe starting with the last, Claudia, and your experience with the IMEDS program. I think you opened the presentation saying that the query, or the initial safety questions that you went to IMEDS with was related to some regulatory activity at the company. So the question is, given your experience in working with the IMEDS group, where do you see the best opportunities for that kind of an approach to support a company? Is it responding to a safety question that the FDA might come to the sponsor with? Or is it embedded as part of a post-market requirement? Or what are some of the, I guess, the best opportunities for that program, for a company like yours? – Well, I think that, really, the size and the quality are the two reasons that would drive, and certainly drove us, to look at the IMEDS data.

So for any rare outcome, you’re looking for a database that is as large as possible. And had we been interested in one of the individual outcomes more than the aggregate VTE, I think that it would have been challenging, even with a database as large as Sentinel’s. And again, having the validated programs that are sort of hands-off, you define the parameters ahead of time, but anybody starting with your parameters is going to get exactly the same results. It is very reassuring, both to industry, and I’m sure, to regulators. So that is another strength of this data. – Great, thank you. One quick question to Robert, with your CNODES example. It’s a two-part question. The first part’s really fast. How long will it take us to get to a point where it doesn’t take two years? So you’re talking about bringing in the Common Data Model to make things more efficiently, how long will it take to get to a point where a query that’s run on the Sentinel System here could basically be simultaneously run on the CNODES program? – So I guess our ambitious target is sort of April, May of this year, but I think this summer is a realistic target to be able to be at an operational stage.

– So my followup related to that is, is it that simple? The data that are used in Sentinel, and as you indicated, for CNODES, are largely claims data. Those are there because insured populations have medical encounters, and one of the challenges with claims is that sometimes the benefit structure of the health plan might have influence into the patterns of care that you see. Should we worry about, or should we spend some time looking at differences in just the utilization patterns between what we see in the Sentinel experiences, versus what we see in Canada? – So the short answer to that is yes, I think we should be looking at those sort of patterns in our experience within Canada, because there’s variation. The systems are similar across Canada, but the specific coverages, the formularies can differ from province to province. So we’ve had examples where utilization patterns for, for example, one of the ones we’ve talked about before, proton pump inhibitors are very different in different provinces because in at least one or two spots, they’re only covered as a second-line therapy after H2 blockers.

So we’ve experienced that, where you’ll see a drug that is very common one place, and one pattern of associations with outcomes in one place, looks completely different in the next province over. And the people are the same, but it’s the formulary, and the programs that are very different. – Okay, Dan? – Hi, I’m Dan Mines from Merck.

– Do we have a, can we get the microphone on in the back? – Hello? – Okay. – Hi, I’m Dan Mines from Merck. Thanks very much to the panel for very interesting presentations. I have a question for Chris Granger regarding, basically, I was really impressed with the ability to do tests of an educational intervention using randomized trial within the Sentinel network. I’m wondering if there are any situations where you could actually envision a trial of studying the effects of drugs using this framework? – Yeah, I think the answer is potentially, but it would, of course, in terms of a pragmatic approach, it would probably need to be commonly used, evidence-based treatments that one might compare. And then one would clearly need informed consent at that time, which would then require selection. But in fact, the ADAPTABLE trial done through PCORI, done through PCORnet, is leveraging some of these same issues, in terms of at least supporting a randomized clinical trial of two doses of aspirin.

Although, that’s through PCORnet. But I think the answer is yes, potentially, but the lower-hanging fruit is these lower-risk interventions. – Where you don’t need to get the informed consent? – Yeah, I mean, I think Rich could comment on that as well. I think it’s been a remarkable, remarkably important part of this highly pragmatic trial, where we can really, where we randomize 80,000 patients at low costs and include all eligible patients. – Great, so I have a question, maybe I’ll turn to Allison. I think you started in the beginning, or somewhere in your presentation you said, when we get UDIs in claims, there would be a big opportunity for Sentinel to help NEST develop better evidence for medical devices.

Short of that happening any time soon, are there opportunities where the Sentinel infrastructure can partner with NEST, or at least partner with registries to help develop better evidence for devices? Do you need to have UDIs in the claims to do that? – Right, yeah, because the adoption of UDI, it’s a big problem to tackle. And it’s taking a lot of work. And there is a lot of great work being done in CDRH on this front, but there are several steps. We’re starting with the GUDID database, which contains devices with their UDIs. And then there’s a lot of work that will need to be done to get these UDIs into the device labels. And then the bigger hurdle into the actual EHRs, claims data. So other strategies we can use, as you mentioned, partnering with specialized device registries, or coordinated registry networks, which provide an opportunity to identify patients who have received treatment with a medical device.

And then through data linkages, with sources of data, such as claims data, such as Sentinel, such as death index data, then provide an opportunity to really look at longer-term followup data for these patients, absolutely. – Great. – And also, I would point out a great example is the TVT registry, which Dr. Pappas mentioned this morning, which is already a great precedent for this model. – Great, I’ll turn to Beth. You spent, in your project, a lot of time looking at the ability for the programs, the modular programs, or some of the Sentinel tools, to apply them to other data sources. And you listed some limitations and some benefits, what would you say would be your biggest, your recommendation for where to prioritize improving the tools, at least the ones that you’ve used? – That’s a good question. I think one of the big hurdles that we encountered was just transforming the data into the common data model takes a fair chunk of time. So if we’re able to have things more easily available, code out there to transform the data, potentially, that would have saved us quite a bit of time on it.

Other than that, it really just is this issue of adaptability, so the more checks that can be worked in there to find ways that, okay, something went wrong, or something’s not working optimally, and that gives you options for making changes at that point. That really is what I would like to see as the top priority. – Okay, any other questions from the audience? Okay, so I’ll keep going. Well, I’ve got one more, and I’ll turn to– – This guy right here. – I’ll turn to Chris Granger from Duke. Okay, so your recommendation was maybe to start, you know, areas where you could sort of do more randomization leveraging the Sentinel infrastructure. Low-risk, where you don’t necessarily, you don’t have to go through the challenging informed consent process. Related to that, I suspect that the first time going through this, you may have identified a lot of opportunities to improve efficiency the next time around. So based on what you learned, you’re learning in this study, if we were to do this more often in Sentinel, what are some of the big areas where you can improve efficiency from your experience? – Yeah, so, and in fact, in the Collaboratory Grand Rounds, we had some discussion about this, about lessons learned.

And they include some things which you might, which are kind of common sense. That includes engagement of the data partners early on to refine the materials, the protocol, the regulatory issues, basically to have the team kind of fully integrated from the beginning. I think things have actually gone pretty fast, in terms of the trial. But it was somewhat behind what we’d originally envisioned, in part related to those challenges.

Some of the key things were like getting Medicare Advantage on board. Rich was able to get a letter from Medicare saying it was okay to go ahead and include Medicare Advantage patients in the trial, which was a key factor. There were a variety of examples like that, that were important. – Great, Rich? – Hi, just two things. One is Sentinel licenses the MarketScan data, and has transformation code. So if you are a licensed user of MarketScan, we’re working through the market with IBM Truven to be able to release that code to licensed users, so– – Thank you, Rich. – Yes, thank you very much. – All right, we’re in the final stages of making that, for good and sufficient reasons, we can’t just post that the way we will post the code for transforming CMS data into the Common Data Model. And then the other point with regard to Dan’s question about studying therapeutic agents, the example Ken Sands gave this morning I think gives some sense of another way you might do comparisons of therapeutic agents through formulary management.

I’m not saying it’s a straight shot, but if you’re talking about preparations that have the same indications, you could randomize formularies. And that, it’s a different study design, it would teach you different things, but it’s a way forward. – Great, thank you. Last question. – Dr. Granger, has Duke, with all its expertise over the years, when you built your own stuff, now with Epic. Has Duke built an app where Duke grandparents with AFib, horrible AFib, can actually say, yes, they’re complying each day, taking their med, but yes, they had an episode, they were just in the ER, because they accidentally took their husband’s pills, and what the patients, why don’t we do that? Why don’t we let people opt in with an app? You know, 85 year old people and 80 year old people actually do use iPhones. And they can say, you know, I mean, AFib is just a horrible illness. – Yeah, great question. And the answer is yes, but not quickly enough. And not extensively enough. So we’ve done a randomized clinical trial, for example, to give healthcare providers best practice alerts when a patient comes in with a diagnosis of atrial fibrillation, and is not on an anticoagulant.

And we’ve had educational programs for patients, but we have not had as comprehensive a program as we should, and we’re actually in the process of doing that. We’re doing a, we’re involved with the Premier Group on a cluster randomized trial at a hospital level of trying to improve anticoagulation use at the time of discharge and a variety of other programs. It’s a key opportunity. I think at the same time that we do this type of program, we also need to be working with health systems in leveraging electronic health record opportunities to better measure and feedback care for this type of a problem. – Are you using the Epic database and data warehouse to actually right now proactively in the Durham Chapel Hill research triangle, RDU, RTP area, are you gonna go actively manage and grab anybody on an AFib med? – Not exactly, but we are, I think that kind of idea is what we should be doing.

And we’re exploring those things, yeah. – I’d be honored to help. – Okay, thanks. – I’m real active with Dean Broome and her nursing advisory council. – Great, thanks for volunteering to help. Okay, so that takes us to the end of the panel. Thank you to all of the panelists and presenters for great examples, very good discussion. We’re gonna go ahead and take a break now, we’re gonna start back in 15 minutes at about for the final session. Thank you. (audience clapping) – You all to head back in and head towards your seats. And if our other panelists could head on up to the front, I’d appreciate that, too.

Thank you very much. We’re about to start this final session for the Sentinel Initiative event, thank you. This group really likes networking together. So the way that. Everyone, I’d like to welcome you back to today’s final session. This is one that intends to build on many of the themes that we’ve heard about this morning, starting with Commissioner Gottlieb’s keynote, as well as some of the ideas that different leaders in different parts of FDA brought up about the future of the Sentinel System, as well as many of the ideas that have come up in the last several sessions, focusing on some current activities, new applications, and potential path forward for many of those. So this is about a look into the future of the Sentinel system. What might the next decade hold for Sentinel, and what are the important steps in the near term to help us get there? So we’re gonna take a look at some of these possibilities from a range of perspectives.

We’re going to have some time for discussion with the group, in addition to their presentations. And as always, looking forward to your comments as well. So we’re gonna start out this panel with Bob Ball, who’s the Deputy Director of the Office of Surveillance and Epidemiology, at the Center for Drug Evaluation and Research. Then Rich Forshee, the Associate Director for Research at the Office of Biostatistics and Epidemiology, at the Center for Biologics Evaluation and research. Alan Williams, Associate Director for Regulatory Affairs in the office of Biostatistics and Epidemiology, at CBER, the Center for Biologics. Darren Toh, Associate Professor in the Department of Population Medicine at Harvard Medical School, and Harvard Pilgrim Health Care Institute.

And the Director of Applied Surveillance at the Sentinel Operations Center. And Josh Gagne, Associate Professor of Medicine at Brigham and Women’s Hospital and Harvard Medical School and an Associate Professor in the Department of Epidemiology at the Harvard T.H. Chan School of Public Health. Next, Joanne Waldstreicher, who is Chief Medical Officer at Johnson & Johnson. And Marianthi Markatou, who’s Professor of Biostatistics and Computer Science at the University of Buffalo. We’re very pleased to have all of you with us. Each of them is gonna start with some opening comments related to this look into the future, and then some time for discussion. So Bob, please go ahead. And everybody can just stay seated for their remarks. – Thanks, Mark, so many of the themes that I was gonna discuss have already been brought up, so I’m just gonna pull together some of the highlights. But there is one topic that we haven’t really touched on, which I’ll also like to mention. So first of all, just a reminder that FDA is committed to improving the Sentinel System through the PDUFA-VI agreement.

And that really, in a very broad way, outlines what we plan to do for the next five or so years. This panel is engaged with the discussions out to 10 years, so that gives us the opportunity for more creative thinking. We heard earlier in Michael Nguyen’s presentation about how we’re using the ARIA sufficiency analyses to help us identify the strategic areas where we need to do our further development. And you can see from Michael’s data that the biggest gap in ARIA sufficiency is in outcome validation. So that’s where we’re going to be focusing a lot of our effort, and I think we’ll see that being done in incremental process. First, starting with the data that we currently have, in the current systems, working very practically on improving traditional chart review efficiencies.

And then moving to other technologies, such as machine learning and natural language processing, using the data that we currently have, but really looking forward to how we really would like a system with computable phenotypes to really operate within the context of Sentinel. The other big theme is signal detection. And at FDA, we always go back to our enabling legislation. And in this case, it’s FDAAA 2007. And Michael talked about the provisions that drive ARIA. But there is another mention in there about a goal of signal detection using population-based data sources. And it’s not quite as explicit as the ARIA requirement tied to the post-marketing study requirement, but it is something that we take seriously. I think we have done some work in this area, within Sentinel.

And many of the questions, the technical questions, still remain. What methods, what data? There’s the philosophical question of data reuse. But there’s more pragmatic questions as well, such as, where does population level data fit in the existing paradigm of the way FDA and the pharmaceutical industries use all sources for signal detection with refinement and validation in population-level data sources, which is really the basis for how we use Sentinel now. So the other pointer that we haven’t discussed today is the role of the Sentinel in pharmacoepidemiological methods development more broadly. I think Rich Platt said that what’s really happened in Sentinel over its original 10 years is we’ve incorporated standard methods in automated tools, and I put in place processes for making that work very efficiently.

So the question is what should we be doing, though, to address the more fundamental problems of epidemiology and bias control? And what new approaches might we consider? And are there clearly agreed upon approaches now that we can implement in more specific processes that would continue to advance our ability to address safety questions? And lastly, the biggest challenge for Sentinel is that all these issues have to be addressed in the context of a distributed network with different types of data and data partners.

And we also have to meet the 100 million life requirement that’s specified in FDAAA, not an easy task. And all the while, maintaining patient privacy and data security. I’ll stop there. – Great, not easy task, but thanks for teeing up these issues for the future. And next is Rich Forshee. – Hey, good afternoon, everyone. It’s been a great meeting so far today. I’m really happy to see everyone who’s here. You’ve heard a number of presentations from other people at CBER this morning that have talked a little bit about how we have some special needs because of the kinds of products that we regulate. So I’m gonna take just a couple of minutes to explain a little bit more about what our regulatory responsibilities are and how they effect the data needs that we have.

So at CBER, there’s three major product areas that we’re responsible for. The first is vaccines, the second is blood and blood products, and the third is tissues and advanced therapies. There’s some unique things about each of those product classes that affect the kind of data and analysis that we need to do. Starting first with vaccines, one thing that’s very different about vaccines is that vaccines are given to healthy individuals in order to reduce the risk of some infectious disease in the future. This is very different than using a drug to treat someone who already has a disease that’s causing them some problems. Also, many of the vaccines are given to children, which have a particularly high safety threshold for.

And we also have a special issue with influenza vaccines, because these are given annually, and they’re given in a very compressed time frame. In a few months, we have about 100 million or more vaccines that are administered. So as a consequence, we have a very low tolerance for any risk for this vaccines. Certainly, compared to cancer therapies or drugs to treat other very serious diseases that people already have.

And we also have a need for very timely data. If there’s a problem with a seasonal vaccine, whether it’s an issue of efficacy, or any safety issue that we may find, we need to know that as soon as possible. With blood and blood products, I just want to mention that, first of all, blood donation is something that’s very common. And you can get many different life-saving products out of any given blood donation. Blood transfusions are really very safe, but there are still remaining safety concerns. One of the concerns is infectious diseases. Certainly, in the early days of the HIV epidemic, that was a critical concern. And also with emerging infectious diseases, we need to be able to respond quickly in that area. CBER is also, make sure that we want to ensure the safety of blood donors as well.

Making sure that they’re healthy enough to donate, that they aren’t donating too frequently, for example. And finally, in the blood and blood products area, we need to balance all of these safety concerns with maintaining and adequate supply of blood to meet all of the medical needs that we might have. Just mention a few things about tissues and advanced therapy, we regulate a wide range of human cell and tissue products. This includes things such as bone, skin, corneas, ligaments, tendons, oocytes, and semen. A very, very wide range of tissue products. And we also are responsible for certain advanced therapies, to give you just a couple of examples. Last year we approved the first two chimeric antigen receptor T cell products. One for treating leukemia, and one for treating large B cell lymphomas. We also approved last year the first directly administered gene therapy product.

And this was treating a mutation in a specific gene that can result in blindness. So I mention all of this just to give you an idea of why some of our needs may be different than needs in other centers. So as a consequence, we’re always looking for new sources of data that help to better meet the specific needs that we have within CBER.

My colleague, Dr. Williams, is going to be talking in his presentations specifically about our new initiative, the Biologics Effectiveness and Safety program. And how we’re trying to use that to address some needs that are specific to CBER. I’ll go ahead and stop there. Thank you very much. – Great, thanks very much. And next is Alan, and I think you need the clicker. – Thank you, so I’m gonna build on some of the introductory remarks made before lunch, by Azadeh Shoaibi, regarding the BEST effort. Particularly the innovative methods work that’s being done.

And I’m gonna approach this from a specific use case related to efficacy and safety for blood transfusion recipients, otherwise known as hemovigilance. For several reasons, hemovigilance in the US, despite some very dedicated efforts through the years, has been very difficult to obtain in a reliable fashion. Several reasons are exposure data are difficult to obtain. I know there has been work done with claims coding related to blood transfusion, but for various reasons, which I won’t go into, estimates in the literature are from 40 to 70% sensitivity of exposure codes related to blood transfusion. Specificity is good, but it’s illegal to not be good at that, so. Codes for exposures remain something that either needs to be improved, or we need to find a better way to go about it. And then outcome data, similarly, are difficult simply because these patients who have received transfusions have many other medical things going on, and it’s sometimes times tough to tease out symptomatology and lab findings that are specifically related to a transfusion exposure.

Why to do this? Not only for surveillance observational purposes, but for quality purposes, the blood community would very much like to benchmark between institutions and be able to evaluate their own work. And also several of these adverse events are amenable to intervention and, of course, intervention evaluation is very important. The numbers are shown here, basically there are close to 18 million transfusions per year, it’s kind of a shrinking industry, because of better blood utilization programs. But to bring it home, probably 40 to 70% of the folks in this room will receive a blood transfusion at some time in their lives. So it’s an important issue to study. Manufacturing steps are many, and some are listed here. Exposure codes, when they are available, typically only list the blood product, and some very basic manufacturing steps, nothing else.

Whereas some of the ISBT codes that Azadeh mentioned, and I’ll go into in a little bit, have much more granularity to them, because they’re actually labeling codes for the blood unit. Most notably, at the bottom of the slide, I’m not referring right now to transfusable blood products, which are derivatives made from blood plasma, pretty much as a pharmaceutical product. So hemovigilance in the US, there are about 14 blood component recipient adverse events recognized. They comprise hemolytic, non-hemolytic, infectious, and unclassified as categories. Severe adverse outcomes are uncommon, but they rarely can be fatal. And when you multiply it by 18 million, the numbers start to look important. The ISBT working party on hemovigilance has put forward transfusion reaction case definitions, which are published. And these have formed the basis for many of the hemovigilance efforts that have been undertaken. Some of the pressures that make it difficult to conduct hemovigilance in the US is, number one, probably, we don’t have a national health system. And many of the things that we’re talking about today is because we don’t have commonality between our data systems and interoperability.

So that remains a major problem. But the blood community is a small industry, limited resources, transfusions recipients often had serious underlying conditions. Each transfusion component, unlike a pharmaceutical, is an individual lot, they’re not made in batch. So each product has its own biologic characteristics. And as mentioned, the availability and validity of exposure and outcome data has been challenging. So the Sentinel BEST program right now has been working, as Azadeh mentioned, to use ISBT-128 blood component labeling codes as exposure variables. These are machine readable, as required by the FDA. And they do provide accurate and detailed descriptions of transfused components. Importantly, these codes, at least in our experience so far, have not been located in the EHR. You actually have to get to the blood bank and download their database to find these codes. And as mentioned by Azadeh, they’ve been incorporated already into the OMOP Common Data Model. The Common Data Model itself, I think, has been really well described, the OMOP model. I think an important thing to mention here is using something like data mining, you often need an iterative process to look at the variables that you describe, see how they play out and modify them.

And a really rapid analyses is important to that process. And OMOP seems to support this very well. Adverse even data is difficult to come by. And the effort within BEST is to use data mining, specifically NLP at this point, to use a combination of text data, diagnostic codes, condition codes, and the combination thereof to define high-quality exposures from the charts. Why is this important? I’ll demonstrate this by showing a couple slides from a validity study that was done by the AABB, which is our major US blood banking professional organization. CDC, the Centers for Disease Control Prevention, and the AABB has been working for close to a decade on a national hemovigilance program.

And this is a voluntary collaborative effort. The design is based around the ISBT working hemovigilance definitions. The study has uniquely has the denominator available, which is a big asset. The problem so far is out of the several thousand transfusing hospitals in the US, only a little over 200 are currently enrolled, and that’s because of some of the pressures I mentioned. That difficultly in generating these data and keeping up with an active reporting system. So AABB conducted a validation study within the NHSN Hemovigilance module that was published in Transfusion in 2014. It used the standardized hemovigilance case definitions. And it was conducted by some of the founders of the NHS and Hemovigilance Study, who assembled 36 fictitious post-transfusion AE cases from 12 different diagnostic categories. These were sent out by survey to 22 different academic medical centers, of whom 50% were participant in NHSN Hemovigilance Program. They asked for case diagnostics as well as parameters related to severity and imputability with respect to transfusion linkage.

And then the concordance was determined by an expert assessor. Interesting that despite this well-controlled situation of active hemovigilance, the case diagnoses concordance ranged from 36% to 72%, and 36% coming from transfusion-associated circulatory overload, which can be fatal. I think that reflects the difficultly in trying to find diagnostic outcome measures which can be applied for any sort of hemovigilance measure. But also, some of the difficulties in assigning diagnostic codes just within a typical medical chart. Interestingly, there we no differences observed between reports from NHSN participants and non-participants. So the training at that level did not seem to make a difference. So within the BEST program, our collaborators, and I wanna mention OHDSI and IQVIA, Christian Reich and Patrick Ryan are here representing both OHDSI, and Christian’s with IQVIA. And we’re working with John Duke at Georgia Tech, who’s actually doing a natural language processing. So we’re attempting to apply these methods to electronic health records, use the mining to extract outcome information from the EHR text, apply it to the ISBT-128 information and try to improve the overall process.

The focus right now is bacterial sepsis, transfusion-associated circulatory overload and TRALI, transfusion-related acute lung injury. One reason is that all three of these are really amenable to interventions that can be measured. So we’re hopeful that what this process, within a short period of time, we’ll be able to move forward and improve hemovigilance overall in the US. And we have, I look forward to extending this to other CBER products in future years.

Thank you. – Great, Alan, thank you very much, that covered a lot of ground there. And next is Darren. – So I’m gonna talk about some of the work that we’ve been doing to try to analyze information from the sample of individuals that have collected into multiple data sources, but the plot twist here is without physically pooling the data sources together. So we all now that our data are now being collected in multiple data sources, and these sources complement each other. If we are able to analyze these databases together, we will be able to get better information and generate more valid evidence. So this is a very typical setting, if you multiple database environment. You have multiple data contributing sites. And one of them can serve as the analysis center. So one way to do this is to ask everyone to create a database and then send their database to the analysis center. In my personal experience, this has never happened before. So we have to go with another approach, which is to try to process your data a little bit further at the site level so that, in the end, what you are combining is a study-specific individual-level data set.

And what we are able to do now is that the individual-level data set can be freed of any identifiable information. So this is a very typical data set that you see in this type of environment. One observation would be one patient. One column will be one variable. So in the setting that we work with, we might be talking about hundreds of thousands of rows and maybe a few dozen or so columns. So the thing that we have to balance is the data that we think we need to do the analysis that we wanted, and what data partners are able or willing to share. So as of today, there are still a lot of concerns about sharing de-identified patient-level data. I won’t go into detail on all of them.

I will say one things that even if data partners are willing to collaborate, sometimes they do have restrictions about the type of information they can share. Maybe because of the contractual agreement they have with the members or the patients that would not allow them to share patient-level data for secondary purposes like public health surveillance or research. Sometimes the paperwork you have to go through might just be too tedious for analysis to be done in a very timely fashion, so we have to come up with other ways to share information. So this is what we are trying to do in Sentinel. So we try to take a step further to process the data even further so that by the time we get the data, it will all be summary-level. In the past workshops, I think that we have talked about some of the analysis that we have completed. So we are now able to do very robust multi-variable adjust analysis using only summary-level information. So how do we sort of use this principle to analyze this newer environment, in which you had two or more data sources and then they contain information from the same set of individuals, but each data source has slightly different information.

So let’s simplify with setting a little bit, let’s say that we have two databases. One is from the claims data partner, the other is from an EHR data partner. You can replace these with any other data sources. The concept is the same. Let’s say that we are able to say that, okay, we have the same set of individuals. And each of these databases sort of have different information about the patient. A very intuitive way, which is still the commonly practiced approach is to ask the data sources to share the patient-level with you, so that in the end, you have what you have in the bottom, a very nice patient-level data that will allow you to do the analysis. But what if data partners are not able to share patient-level data? Can we still analyze the data to get the evidence that we wanted in that setting? So this is where some of these newer methods come in. So the approach that we are developing or refining is actually a suite of methods called distributed regression.

So at a highest level, it is just like any ordinary least square regression analysis that we are used to doing with patient-level data. But again, the trick here is that these types of methods can do the analysis without actually pooling the databases in the centralized location. And the way it does that is by sharing what we call intermediate statistics, that you are going to see in a minute. And based what people have written down on paper, these methods do follow the same computation pathway as a typical patient-level data analysis. So in theory, this should give you identical results to what you will get from a patient-level data analysis. So this is what I mean by having the same computation process. So many of us run regression analysis using statistical software like R, or SAS, or Stata, and we almost always input a patient-level data set.

And there are things that go behind the scenes that we don’t see. So if you use SAS, once you say I’m going to run PROG REG, for example, using a patient-level data set. SAS or R, they do something behind the scene that we don’t see, and one of the things that it do is to create some sort of summary-level data set that you are seeing in the middle of this figure. What we usually see is something on the right-hand side as the final results. Everything that goes behind the scene is hidden from us, but for us this distributed regression methods, there are trying to actually open the black box, if you will, by specifically using this intermediate statistic. And the nice thing about it is that it is not patient-level data and it is less identifiable.

But at the same time, it follows the same pathway in the computation process. So you should get identical results. So instead of sharing this, what we are going to ask the data partners to share, either with each other, or with a third like an analysis center, is something you see at the bottom, which is just summary-level information. So this is some of the work that we are doing. So it’s just from any of the databases that you could use. So these are the result from the traditional patient-level data analysis. These are the result from the distributed regressions. So these are the numerical differences. So you can see that are quite similar, different maybe at the fourth or more distant decimal place. It is possible to get even closer results. So in some of the settings, we were able to get results that were identical to the 16th or 17th decimal place.

So for all intents and purposes, they are identical results. But distributed regression do this type of analysis without patient-level data. The reason why you have not seen these methods in real life is because to do this type of analysis, you actually require multiple iterations. So the information would need to be transferred back and forth between the data contributing sites and the analysis center. This is just the way that regression models work. So the thing that we have to do is not just to do the analysis, but also to allow these type of information exchange to be sort of semi-automated, or fully automated, so that it is practical in networks like Sentinel or PCORnet. It is possible to do this manually, but you have to people sitting in front of computer to do this back and forth manually, and it’s just going to be too tedious for this to be practical. So were are developing this automation, in PopMedNet, which is the data sharing platform that Sentinel, and PCORnet, and other networks are using. So this is where we are. So in terms of the statistical code that is doing the computation, we have some functional prototype in both R and SAS.

We are continuing to include more secure protocols, so that it’s less identifiable and more privacy-protecting. And in terms of communication code, so we also have a functional prototype in PopMedNet that will allow the user to specify different levels of automation. So it can be fully manual or completely automated, depending on their preference. The final thing I will say is that for all the linkage work that you have to do, you need a global key, which is an ID or some sort of hashed ID that you would need to say this person in this database is the same person in that database. There’s no way to get around that. But for distributed regression, that will be the only piece of patient-level information that you will share. You don’t need to share anything else. And it’s sort of doing this on a virtual linkage to do the analysis that you wanted. So in conclusion, we think it is possible to use these methods to analyze data from multiple sources. And we are trying to reduce the barrier to data sharing by using these newer privacy-protecting methods, thank you.

– Great, thanks very much, Darren. And next is Marianthi. – I thought it was Josh. – Josh. – Oh sorry, next is Josh. It’s Josh, I’m sorry, got ahead of myself. (chuckles) Thanks, Josh, please go ahead. – So I’ve been tasked with, in 10 minutes, giving you a very high-level overview of the rapidly expanding field. So sticking with the theme of better, cheaper, faster, I’m gonna go quickly. I will say that if my flight back to Boston is canceled, I’m happy to stick around and provide more details on any of this. It is high-level, everything that I’ll present is in the public domain, references are provided on the slides, if you wanna take a deeper dive on anything.

So I think in thinking about how to distill this into 10 minutes, a good place to start is this recent review paper that came out of Bordeaux that attempted to classify and describe the whole host of signal detection methods that can be applied to the types of longitudinal data that we’ve been talking about today. I’m not gonna go into all the details of this, the tables here are just to kind of show you that there are lots of different methods that have been proposed, used, and described in this paper.

They are very heterogeneous. So some methods are your typical pharmacoepidemiologic study designs, cohort-type approaches, self-controlled designs. Others are methods that have been borrowed from the spontaneous adverse event reporting world and brought into longitudinal healthcare data. And then there are some that I wouldn’t necessarily classify as being signal detection methods inherently, but they are approaches to doing multiple testing, either sequentially over time, or across a large space of potential signals. And one of these methods, which is at the bottom there, for anyone in the front who can see it, is the TreeScan approach, which I’m gonna come back to in a few moments. When we think about the signal detection methods that we could apply to electronic healthcare data, it’s important, I think, to start with asking what we’re asking of the methods. And fundamentally, we’re asking them to screen potentially many drugs and potentially many outcomes to tell us what are signals that merit further attention. And these are, the opportunity for signals, when we think about how big the Sentinel data is, and we heard Rich earlier today say that the Sentinel distributed data network is nearing 300 million patients, with the addition of the Medicare data.

These big data, and when we talk about big data, Sentinel is big data, represent a huge opportunity for false positives. So the signal detection algorithms need to tell us what to look out for, while at the same time, trying to filter what are the true causal relations, those cause and effect signals that we care about, from signals that are arising from chance or from bias. And chance and bias are things that we deal with every day, in pharmacoepidemiology, not just in the signal detection framework, but also in any hypothesis-driven study that we do. And as such, the methods to filter out chance and bias are very similar to the methods that we might think about in hypothesis-driven studies. They’re epidemiologic methods, they’re statistical methods, and they require, as well, clinical input. In thinking now about how to bring these methods into a distributed data environment in big data, I like to think of the methods as having two fundamental features that are sort of core to an effective signal detection algorithm.

One is akin to the design. So all of these methods, in order to be effective, are producing observed and expected counts, which is sort of the basis of epidemiology. We’re always comparing observed and expected in some way. And it can do that via any design, the cohort approaches, the self-controlled approaches. And again, I think of this aspect of the signal detection methods as the design component, which is really intended to filter out those signals that could be due to bias. The second key component is some process to then prioritize or rank signals in some attempt to separate out those signals now that are due to chance. And there’s lots of different ways of doing this. And I think of this as being akin to the analysis component of a study, where we could use an alpha-based approached, and the TreeScan approach that I’ll talk about uses this within a frequentist framework, to tell us what our statistically significant signals, while adjusting for all of the multiple testing that’s happening.

But there are other approaches as well. Bayesian approaches, and there are approaches that should and can consider clinical factors that should determine what signals we should prioritize for further followup. Or biological plausibility as well. And I think those factors shouldn’t be lost, as we’re thinking about this. Now going into the data, we have these methods. And there’s a couple of different paradigms that we can think about in casting these methods within the data that we’ve been talking about today.

And I think that these paradigms exist along a continuum, where we’ve heard a lot earlier today about more targeted pre-specified analyses, where we have a single drug or medical product and we have a single outcome or some finite set of outcomes that we’re interested in, in specifying and evaluating. And this, for the most part, is where Sentinel has focused early on, and in particular, with Mini-Sentinel. That’s largely where the activity was. There’s also at the bottom what I would call a completely hypothesis-free space, an all-by-all type screening approach, where you could, in theory, think of an approach that would screen every possible medical product exposure and outcome, and do this all-by-all activity. I’m gonna actually, I have a couple of slides here I’m gonna just fast-forward through, just because I think there have been a couple of examples of targeted analyses that have been presented throughout the day. And since, I wanna make sure we have plenty of time for discussion, I’m just gonna quickly go through these.

But these are examples, that again, are in the literature, and also on the Sentinel webpage, that have done the targeted sort of assessments, both in a retrospective as well as prospective way. And where Sentinel is sort of moving in the signal detection spaces now, thinking about how do we do this kind of middle activity, where it’s not quite the all-by-all screening, but it takes either a drug or outcome perspective.

So you might start with a single drug and say, is this drug associated with any potential outcome that we can observe in the data? So it scans across all potential outcomes for a single drug, or you could flip that around and say, we’re interested in one very specific outcome, can we scan across all drugs to determine whether there’s any specific drug or drugs that may be associated with that outcome? And again, this is kind of where we have focused more recently in Sentinel. And one way of doing this, and this was on that table from the review paper that I started out with, is this TreeScan methodology. And I’m gonna give you just a very high-level overview of what TreeScan is, but to the second bullet point, one of the really appealing features of TreeScan is that it’s agnostic to where we get those observed and expected counts.

So it’s compatible with essentially any design component of a signal detection that we might consider, with cohort approaches as well as self-controlled approaches. It uses for, what I called the analysis piece, or analysis component, a formal multiple testing framework to account for the fact that it’s making repeated tests across lots of different outcomes. And as you’ll see in a moment, across lots of different levels of outcomes. So it’s adjusting for this multiple testing, and in a very formal way that not all signal detection methods do. And then at the top, the third important feature of TreeScan is that it uses a tree-based structure to do the scanning across outcomes. So I’m really focused here on that approach in the middle that focuses on the drug perspective, taking a single drug or medical product and looking across multiple outcomes. Here’s one example of what a tree might look like, across which we can scan. So this is the MLCCS structure, it has four levels. At the top, there are 18 body systems.

And then all the way down at the bottom, at the fourth level, and you don’t need to see the detail here, it’s the sort of structure that’s important, are our ICD codes that are then what we see in the data that we usually use to measure our outcomes. And they are rolled up into these higher-level terms. And then further rolled into these body systems. And what TreeScan does, at a very high level, is it allows you to test both across all of these nodes at any of these levels. So if you wanted to test across all fourth level nodes, you could to that. It also tests across each of the levels. And so it does this in a way that it counts for the corelation in these outcomes across these levels, and also the multiple testing that’s happening to preserve the alpha. I’m gonna briefly talk about what we’ve been working on, and when I say we, I’m saying that generously, to myself.

There’s been lots of people who have been leading this, and I’m summarizing this work for them. There was a pilot study done looking at the TreeScan approach with a self-controlled design. Specifically, a self-controlled risk interval. And this is one that Katherine Yih led. And it looked at the HPV4 vaccine, or Gardasil, and scanned across million doses of the vaccine and found only two signals. Again, the opportunity for false positives with almost two million vaccines is huge. And only two signals arose, both of which were explainable, because they were known and labeled. Either for the HPV4 vaccine, or other vaccines that were co-administered on the same day. There’s now some work that Judy Maro has led that’s wrapping up, that’s taken the TreeScan approach, similar to this with the self-controlled design, and moved it into the drug space. So there is a project, again, that’s just finishing, that was looking at three example drugs. Again, using the self-controlled design. And the results of that work are forthcoming. Often when we do analyses of drugs, whether it’s in signal detection or more kind of hypothesis-driven analyses, we like to think of using cohort designs, ’cause that resembles the randomized trial that we might otherwise have done if that were feasible.

And so there’s been some work done now, including work that Shirley Wang has led that has looked at how we can use TreeScan in combination with a cohort-based approach with propensity scores to deal with measured confounding. And this first project that’s also just wrapping up looked at doing this within a simulated environment. And so this was largely a demonstration of efficacy that this could work, TreeScan with a cohort approach specifically for drugs. And then the next step will be to examine this in more of an effectiveness framework. So looking at actual empirical examples, similar to those that I mentioned that were done for those three drugs for the self-controlled design. So to wrap up, I wanted to highlight three key issues that I see as being important challenges that we have to tackle, not just within Sentinel, but as a field if we are to push the envelope on doing signal detection in a meaningful way. One is that we’re now, in Sentinel, and we’re broadly thinking about how to use TreeScan in these signal detection methods in a cohort framework where we have every possible outcome as an outcome of interest.

And we know that confounding is outcome-specific, so we literally have to come up with a way to deal with confounding for every single outcome. And this isn’t easy, but I think we have some good ideas about how this could happen. Secondly, I think an important way of bringing this back to this framework of paradigms that I outlined is that we should also be thinking about how to do this in a prospective way. So as new drugs come into the market, we can do this TreeScan or signal detection type approach prospectively to identify any serious concerns early on. And then finally, Bob mentioned at the start of this session that this issue of data reuse is an important one. And it’s an old issue that continues to be debated, and rightfully so, it’s a complicated issue that has lots of differing opinions. And it’s a challenge that we need to think through about how to prudently use signal detection methods within these big data systems. And I’m not gonna go into details on this unless others want to, but I just want to leave you with those thoughts, and looking forward to the discussion.

– Great, Josh, thanks very much. Again, you’re all covering a lot of ground. Joanne? – Thank you so much. And thank you for the opportunity today to provide comments. You know, Sentinel has been such an important contribution to public health over the past 10 years since it’s been created. Especially as we’ve been talking about today, to answer really important safety questions. And with very high-quality evidence and data. I think it’s, you know, you asked to think about the next few years and also the next 10 years, so I think it’s really great. And we’ve heard today that the next few years will see some more automation, thinking about AI, machine learning, natural language processing. I think that’s incredibly important and will add a lot of value, especially with the goal that we’ve heard a few times today of better, faster, cheaper.

And I think that Sentinel has achieved that, and I think that there’s more to come with a lot of those new frontiers that we’re all getting into. I think as I think about the next 10 years, I think about the mission of Sentinel. And the critical importance that safety has played, but I also think about a learning healthcare system, which I know is mission that all of us are engaged in. And I think of the important role that Sentinel could play given the expertise, the data sources, and all of the new developments, and protocols, and programs, and contribution to science that Sentinel has and will make.

And what it can do for a learning healthcare system of the future. So just thinking about that, as I go through my comments, first of all, we talked about additional sources of data. And I know that that will be one of the focus areas for Sentinel. And one of the things we might think about in the future, again, I’m talking about the next 10 years, is what we now call kind of patient-centered data. And I know most of you probably saw the announcement from Apple that they will be having one of the apps on their phone aggregating patient-level data. The patients’ own data from multiple sources, aggregated. And there’s also another company called Hugo that is already doing that, I have the app on my phone. And really, the current electronic health record system that we evolved into in the United States really enables that, so that patients can see all of their data, irrespective of the healthcare system, or where the labs are done, or what hospital they happened to be visiting.

And I think that will empower and enable research in the future focused on the patient, irrespective of where that patient is treated. So I think that might be a frontier for Sentinel to consider in the future, as another source of data. Second, new types of studies. And I was very excited to hear about Sentinel getting into cluster randomized studies. And especially the IMPACT-AFib study, which I think is extremely impressive. And just to think about how quickly technology is going, you’re using patients who are treated with AFib already, that have that in their record, and you’re reaching out to them.

Well, we all know that it’s not the future, but now patients can self-diagnose AFib, right, by putting their fingers on a new device that’s coming out on the iPhone or on the Android. And so this opens up a whole new level of research really centered around patients and their own diagnosis via technology. Okay, so in addition, with all of the tremendous work that’s been done with Sentinel, to share some of the programs, and the work, and the best methodology that can be used with those outside of Sentinel. And I think to really leverage and amplify on the learnings that have been gained with the work in Sentinel, and with others who have worked with FDA, and successfully navigated observational research protocols. I think it would be great if there were kind of a living library or a catalog of sorts that could be used to give guidance, with all of the caveats that every new study, every situation is different.

But when there is a data set, or there are outcomes, that have been vetted and approved through Sentinel, and approved by the FDA for use, or through other organizations, other companies, that have had something vetted by the FDA. It would be great if there were a living library that could be used and continually added to, so that when people start new protocols, they could start from outcomes and other parameters that have already been approved by the FDA. It could be a great public resource. And it could help both researchers, companies, as well as FDA, because there would be less back and forth and negotiation on those kind of parameters. Let me shift a bit. So from Alan’s talk, and from Steven this morning, I am encouraged to see FDA actively applying multiple alternative strategies to meet their needs, including the use of the OMOP Common Data Model on OHDSI as part of CBER’s BEST program.

And I think that development of new methods in BEST in the area of natural language processing and machine learning could be very valuable into the future. I’m also very excited about Josh’s presentation. And to see that Sentinel’s exploring innovative approaches for signal detection and signal generation. I wanted to stop and say that Dr. Gottlieb started the whole day today, and I wrote down his quote, saying that in 2003, he already wrote, for you, Mark, that, “Is there a way we move away from MedWatch?” And that was in 2003.

And so the answer, of course, right now is no. There is, for serious adverse events. But I do think with the approaches that have been presented today by Josh and some other work that’s going on, even in our shop in J&J, with OHDSI, et cetera, there are methods that will we need to work together on to be able to do better signal detection in the future using real-world data, the real-world data. Rather than under-reported spontaneous adverse events, which, as we know, is the basis of our signal detection strategy right now. I actually feel very strongly that we need to start working towards this very quickly. And doing this together. And there are many reasons why. The first one is the most important one, and I don’t have to tell this audience, it’s the public health perspective. But second, the data, in general, big data, are becoming widely available, whether it’s EHR data, PatientsLikeMe, as I talked about, apps that diagnose AFib, et cetera.

And we are not the only ones looking at this data. There are technology companies that don’t have the expertise that’s been built over the past 10 years, who are analyzing these data. But I think it would be best if it’s the group here, in this room, the brains in this room, and all of the people who have been working on these types of data over the past 10 years, to work together and understand the best possible approaches for signal detection. Rather than tech companies doing this in isolation. Finally, I think we do need to think about efficiency. Sometimes, we need to think about what we can we stop doing, and what can we do more of? Some of you have hear me talk about this already, so I know I’m saying something that is not very popular, but can we look also at non-serious adverse events? Which actually take a tremendous amount of resources, hundreds if not thousands of people’s worth of time and millions of dollars.

Imagine if we could have a more efficient way to look at real-world data for non-serious adverse events, rather than the way we look at now, which involves reporting, documentation submission, quality analysis, interpretation, et cetera. And maybe we can learn from non-serious adverse events and that will give us a whole nother way of understanding how in the future we can look at serious adverse events. And this is where we do have some work to do. And of course, we cannot stop doing adverse event reporting for serious adverse events. So let me just close by adding one more thought and one more point. And that is something that Darren Toh touched on. And highlighted a very interesting way to do analyses across a data network without actually sharing patient-level data. And this raises the whole point of the issues that have come up with sharing, or sharing access, to real-world data.

I think it behooves us as a community to look at some of the lessons that have been learned with sharing clinical trial data, which went through several years of very tough discussions, With data holders, just as have data holders with real-world data, there are certainly, of course, data holders of clinical trial data, including J&J. And many years of discussion and negotiation, and a really excellent IOM, or as we say, NASEM, report came out which really outlined a very reasonable path to what they call responsible sharing of data, which could maximize the benefits and minimize the risks of data sharing. And there are a lot of lessons to be learned from those models that were laid out.

And from where we are now, where data are shared. But in several different models. There are models where data holders actually share their data, they give it to other researchers to analyze. There are models where some researchers put their data up on the internet. But there are also models which I think would be very amenable to the governance that we’re all talking about here, where it’s not the data that’s shared, but it’s the access to the data that’s shared. And there’s governance structure. There’s an independent reviewer, for example, to be sure that the researcher who wants access to the data has a good research protocol and is qualified researcher.

I know there’s a, we’re involved in J&J with my colleague, who’s here in the room, Joe Ross and Harlan Krumholz. And their data sharing initiative, which is called YODA. Again, it’s sharing the access to the data, and Yale reviews all of the research proposals that come in to be sure that there’s good governance over the data and the analyses. Finally, one last benefit where I think the Sentinel, Sentinel network, and all the expertise would have tremendous value in the learning healthcare system is really on predictive modeling.

And feeding information that’s gathered from Sentinel into the healthcare system, and then following up to look at the impact. I loved the presentation today looking at morcellators. And what happened, and being able to follow that the changes that the FDA made to the labeling actually resulted in decreased use of morcellators. What I would love to see, in a real learning healthcare system, and I think Sentinel could really help in the future, and I’m sure it cannot be done with the morcellator, because it’s such a small incidence. But imagine, with the morcellator, identify we didn’t just look at the incidence of that type of surgery, but we also then looked at, did we actually decrease the risk of developing sarcoma, of sarcoma, let’s say, in women, in the United States. Did that go down? Did the risk of infections go up? Did the risk of VTE go up? Those were all issues that were raised when the discussion on what to do with the morcellator issue was being vetted. And we still don’t know the answer to that. But in a learning healthcare system, that’s the benefit of doing the analysis, looking at the impact in the healthcare system, what’s going on in medical practice.

Seeing whether we had the right impact, and then continually feeding that back to the healthcare system, so it’s a learning system for our public health. Thank you so much. – Joanne, thank you very much. (audience clapping) Really appreciate the focus on the long-term vision here. We’ll come back to that in the discussion. Marianthi? – Yes, I would like to begin by thanking the three speakers for the very interesting presentations that they gave, and to actually say that I am really impressed by the amount and quality of work that has been done in the Sentinel System.

I come from an academic perspective, and I would actually draw on the common themes that I saw in the three presentations. And I will take it a step further, in a plea of actually asking for being able to look, the Sentinel can look into real new development, development of new methods. A lot of discussion was done, and a lot of around the data, and data sources, and of course, it is obvious that data is not only numbers, like we knew it before, but it involve text, it involve images, laboratory data, socioeconomic data.

Data come form very different sources with the idea of actually supplementing the profile of the patient. So combination from data sources brings up different issues of scale and usability, and development of new methods as well. I was really excited to see the presentation on blood transfusion, actually, using text data. Computable text is really important, and offers a very important element in supplementing the laboratory characteristics of a patient. In terms of other aspects different from the other talks, it is very encouraging to see TreeScan methods that are very well-developed to actually be used in from a specific to broad setting. The setting in which you’d use multiple drugs and multiple adverse events is really interesting and important, as well as the privacy and confidentiality aspects that were brought by Darren’s talk into the forefront here. And with that, what I would like to do is to put in context to what I am saying here, which is when we look into methodological research, there are two broad definitions.

One direction goes into transfer of technology from different fields, and perhaps refining these technology. And the second is development of fundamental ideas and methods that actually shed light onto things of importance. And then they can be fed back into the scientific enterprise in order to actually do better adverse event identification in this particular context. Both of these directions are extremely important. Both of these directions actually offer quite a bit in the quest of delivering better patient care, and in the quest of actually having a learning healthcare system. What I would like to bring in the front here, in order of discussing in the concept of evidence is the development of new methodology in terms of quantifying bias.

There were simple examples that I have put here, in terms of papers. One was a paper came out from the current OHDSI consortium. And the second came as a rebut to this by the authors Gruber, and Tchetgen, Tchetgen. And basically, these indicate a scientific discourse that really sheds light onto the different issues that are at the forefront of our concerns. The second thing that I would like to bring back in the interest of time is other work here. But I’d like to put up this, a rendition of the evidence generation pyramid.

And everybody knows this thing, but I would like to single out, to give you an example here, and single out one specific layer of this evidence generation. It was shown that amiodarone and sofosbuvir are dangerous, they are two drugs that they have dangerous drug-drug interactions. And these are used in hepatitis C virus infection. And for those of you who don’t work in the field of hepatitis, there is some information that I have put down here. But my point is that the FDA put out a modification in the label being based on some case reports that they were actually reported to them. So the additions were made to warnings and precautions, and these were identified, these dangerous drug-drug interactions were identified from using the case reports.

So here you see case reports on the slippery slope. And my question is do they really need to be at the bottom of the pile? And if you are to use them and make inference on the basis of those, how would you do that? So what I am actually proposing here is this idea that that is being used, of the idea of the pattern, the pattern identification.

The pattern here, of course, points to adverse event. There are steps, different steps, for pattern identification and validation. And so that kind of, almost the bottom of the evidence generation pyramid really offers important and interesting information. And so my questions here is can we quantify systematic errors as an alternative to seeking to eliminate bias? And what are the important components contributing to systematic errors that should be assessed? And with that, I would like to stop. And I would like to say that this audience here has the know-how, the ability, to address these and more important questions in the quest of delivering better patient care. And also better adverse event identification. Thank you. – Marianthi, thank you very much for that thoughtful presentation. And I agree with your final point, that between the panel and the audience here, and all of the very broad range of collaborators in the Sentinel Initiative to date, that there is a lot of potential for addressing future opportunities like the ones this panel has raised, and like the ones you all have talked about all day today. And we’d like to spend a little bit of time on that now.

And it includes comments for, questions from you all, relating to where Sentinel should go next. You’ve heard from the panel about both some specific short-term steps. Opportunities and challenges, as well as a long-term vision. And I think we at 10th anniversary, we are at a stage where we’re thinking about the next generation of Sentinel, building on all of these achievements and a trajectory of progress. It does remind me, back in that, now several times referenced 2003 speech, where we did begin trying to talk about that the need for an active surveillance system and focusing on safety. But it was really about using real-world data more comprehensively in FDA’s work. We weren’t the first ones to think of that, nor the last. But the vision there was important. By the way, I’m not sure that replace MedWatch made it into the final version of that speech.

The point was just that there was a lot of need for augmenting some of the limitations of MedWatch. It does seem like now is a good time for more about that vision for the next 10 years. And how it can connect to some of the short-term opportunities. So I would like to talk a little bit about that. And maybe picking up on a few key themes that have come up today. And maybe, again, would encourage contributions from those of you in the, who are joining today, about this topic, but let me highlight three things. One is kind of next steps and a longer-term vision on data. A second is next steps and longer-term vision on methods, particularly around signal detection and effectiveness, not just safety. And third is how this approach that FDA has taken, that I will tell you is impacting, and has impacted, analytics and the capacity to develop evidence from big data elsewhere in the healthcare system. What are the next steps for Sentinel helping to drive us to a true learning healthcare system? There’s been a lot of discussion of big data in recent years, but probably that should be more focused on what you can actually do with it.

Can we actually turn it into big evidence that really is useful for improving the health of our population and people around the world? So back to starting with data. A lot of discussion today around the concept of, Bob, you used the term computable phenotypes. Sort of richer ability to describe particular kinds of individuals and what happens to them. Key to that is the types of data linkages that came up in several presentations this afternoon, Darren, Alan, you all talked about this.

So maybe you could start with a question of whether there is more that could be done on supporting the kinds of data linkages that you described. There’s been a lot of talk, for example, about using hash methods, maybe more generally. Darren, than you talked about in your approach, also concepts like blockchain, somebody had to talk about it today, I guess. But that could potentially permit more use of truly distributed data systems. Alan, you talked about bringing together a lot of different kinds of data with natural language processing. Any further thoughts from you all, or others on the panel, about how to accelerate progress on linking data together to support Sentinel in related activities? Bob? – I’ll start. So whenever I ask this question to Rich Platt, he reminds me that the 10-year vision is actually very easy. It’s all data is linked and then we just analyze it. But it’s the incremental step we take to get there that are important. And since our focus is very much the sort of what we’re doing in the next steps to meet our PDUFA VI commitments in this regard, I’ll just briefly talk about that, and let others maybe talk about some of the more visionary things.

I think one of the complexities that I alluded to in my opening remarks was that doing this in a distributed network, where there’s lots of different data partners with lots of different access to data. So we have integrated data systems that have EHRs. Well, so that’s, they’re already linked. But what do you do with the claims data systems that have to reach back to the medical records, maybe are electronic, maybe they’re not electronic. So there’s a lot of pragmatic steps and we’re working very hard to make that process more efficient. But then I think, when you get, what you can, in the short term, start doing some more of these modern technologies. You can’t do NLP unless you have machine readable records. So can you take those paper records, or the electronic records that are available, and apply NLP to them in some kind of very limited way, initially, maybe to extract some pieces of information before doing full computable phenotypes.

We have done some pilot work doing that. And then, but with machine learning, we could take a corpus of charts which have been expert adjudicated and classified and then seeing if machine learning or other technologies can do a better job with the data that’s in the common data model to predict that classification. So there’s are some of the sort of next steps that we’re thinking of. – Great, thank you. Other comments? – Yeah, I’d just like to build on some of the things that Bob has said. Because actually our two offices have been working together on some of these projects. We have had some real success within Office of Biostatistics and Epidemiology using natural language processing to help build tools to help our medical officers review the large number of adverse event reports that come in more efficiently. So using the NLP as a first step to pick out some of the important elements within the narrative portion of adverse events.

And so I think there is real promise in terms of being able to use some of those methods in other areas. And I liked Marianthi’s comment about computable text being a concept to think about here. And again, I think Bob has done a lot of work on innovative ways to help develop and improve case definitions. My office has collaborated on some of that, and I think that’s very promising. – Any other comments on this topic, yeah? – One thing that I’ve started to think about, and I’m really quite new to this whole area of natural language processing, is that trying to back into it by reading key terms within the text is a little different from having an attending physician that’s been treating a patient and says this is this, I saw a case of this two years ago. This is what it is. You’re missing the physician judgment in doing the NLP process to the extent that it’s not documented that way in the note. So that’s one of the potential hurdles in being accurate with that method. – Yeah, so great comments. And I want to come back to an issue that, well, both Bob and Joanne, maybe put them together, short and long term, the idea, Bob, is to get to having the data that you need to do sophisticated computable phenotypes with reliable validated outcome measures.

And then just run the analysis. We’re gonna come back to the analysis in a minute, but we’re clearly not there yet. And I wonder, in light of Joanne’s comments about moving from sort of effectively a provider-owned fragmented data towards more patient-controlled data, that is a long-term trend that many, including Apple and others, see as part of the healthcare system of the future. But to your point, what are the short-term steps that can help get there? We did have some discussion today about some patient-generated data and some comments from people here about how patients could get more actively engaged in contributing to Sentinel. Are there some short-term steps that would help in this direction? I guess just one more editorial comment here.

In putting data together, there are all kinds of technical issues around linkages, linkage methods to protect confidentiality and proprietary concerns, validation issues, a lot of things in the technical category. But there are also another set of issues which I call governance that came up today, too, about just encouraging and having the support for getting this done. And it does seem like getting patients even more engaged directly in these efforts might help with that. Any thoughts about initial or next steps on getting more patient engagement, more patient-generated data? – Well, the way that patients now have access to their data electronically, right, because the changes in the law and everything we’ve gone through, through EHR. That’s been one of the major benefits that we have. And that really enables Apple, and Hugo, and others to aggregate data based on a patient. And so a short-term thing that we’re looking at is, just like when you accept FitBit, you say yes to everything.

Most of the time you don’t read it, but hopefully you do read it sometimes. So if you do bring all of a patient’s data together, and a patient is aggregating his or her own data, you could ask them, are you willing to contribute, and Hugo does this, are you willing to contribute your data to research, or advancing public health, or whatever the right appropriate approval is. And if you do that, the patient is basically volunteering to have his or her data used in a passive way. They don’t have to keep consenting, right, it’s a passive thing. Their data could be aggregated. So it could be very quickly, I don’t know how quickly these things will get adopted. But based on some of the trends that we’re seeing, it could be that very quickly, a lot of people do consent to have their data aggregated and used for the things that we use the data holders for.

The data companies, this could be another source of data. Of course, with a lot of overlap, right, ’cause all these people are in health systems. But this could be a source of doing studies, doing observational research. – Other comments? It sounds like a good vision. And I would like to, yeah, I do wanna open this up to comments here, so please do come on up to the microphone. – Mine is particularly directed. I mean, part of the question you’re asking right now. Which is, I wonder how many data partners have been considering revising their consent agreements with their patients. Because more and more patients are saying, yeah, take my records and use them. Are data partners thinking about institutionalizing that request? And letting patients opt in or out right there, when they’re signing up to use that healthcare provider? And so just take the idea and run with it, if your institution isn’t doing it already. – Thank you, thanks for the comment.

We would like more like that, here and afterwards, as the Sentinel work goes forward. I’m gonna switch gears a little bit to methods. So assume we keep making progress on these data reliability and linkage issues, we had a lot of discussion today about signal detection. And if there are any further comments on that, based on what you’ve heard today, I’d like to hear them from the panel. But also, this notion of going beyond detecting safety signals, maybe eventually, at the sort of all-by-all, if we can work out the methods issues that have been raised there. But also questions about effectiveness. And those have typically been harder. The effect sizes that you’re looking for are typically smaller and may be a different way of looking at outcome or effectiveness benefits verus safety questions so far.

There have been, as we’ve talked about today, a number of applications of the Sentinel Initiative to effectiveness studies. Often these need randomization. And there have been steps taken to do that, cluster randomization and so forth. I’d really appreciate the views of the panel about further steps that could help encourage appropriate and potentially very useful studies using this kind of infrastructure for questions related to effectiveness. Maybe a label extension study, there’s been a bit of that done already. Maybe other types of applications. Any thoughts, please go ahead, Alan. – Rich. – I’m sorry, Rich. – Yes, there’s a mess up on. (laughs) So yeah, I can speak a little bit to some of the effectiveness studies because my office has done some work on vaccine effectiveness studies using data from the Centers for Medicare and Medicaid Services. And we’ve got a number of these that have already been published, if people want to reach out to me, I can share that information later. But we have done some work attempting to look at the relative effectiveness of high-dose influenza vaccines versus standard dose.

Just a little bit of background, several years ago, an influenza vaccine with four times the antigen level was approved for use in people 65 years or older. And so we have done some studies trying to generate real-world evidence about whether that is more effective at preventing flu. But more importantly, trying to get at some questions that would be difficult to answer using randomized studies. So for example, one of our papers looked at hospitalizations, and another looked at mortality following influenza to try and determine whether we did have greater effectiveness. The results have varied a little bit by season, but certainly in some seasons, we have seen greater effectiveness in the observational data for the high-dose vaccine.

And we’ve also published a paper looking at the, how long the Zostavax, herpes zoster vaccine to prevent shingles, how long that effectiveness lasts. So I think we’re in the early stages of figuring out how to use these studies for effectiveness. And also how to then take the data that’s generated from those studies and use it in our regulatory framework. But I did just want to mention that we have some early experience in trying to do some of those studies.

– Mark, I just wanted to add one thought. Which is that you mentioned pragmatic trials, and the opportunity to do those within Sentinel. We’re involved in our division outside of Sentinel, in activities trying to link randomized trial to claims data. One of the opportunities is that you have longer followup for patients beyond the trial duration in the claims data. One of the big challenges is that for any trial, a small proportion come from any one data partner. Any one insurance system. But imagine doing that in Sentinel, where you have so many data partners that cover a large portion of the US population. There, I think we could do some really interesting work around effectiveness, right. Because if the treatments are randomized, then we can follow them up using Sentinel-like data, we could think about other outcomes that weren’t examined in the trial to really use claims data, in this case, to look at effectiveness in ways that would be difficult in just an observational study work. Confounding for effectiveness might be much stronger than it is for safety.

– All right, thanks, Josh, for that. Bob? – I wanted to comment on the signal detection part of your question. – Okay, that’s fine, too. – And I wanted to refer to Marianthi’s presentation, and her comments about the case-based approach and pattern discovery. And I referred to using machine learning from the, with a essentially supervised machine learning, to identify cases, but I think what Marianthi is referring to, and I’d to maybe ask her to comment on it, is a very different way of thinking about how we do this signal identification, where most of the methods we rely on now rely on very specific codes in a coding system, or the TreeScan method relies on the tree structure of the ICD-9 codes. But there’s a, I think what Marianthi’s referring to is a more general statistical approach, that would allows us to identify patterns in the codes, and really open up the idea of signal identification. And not have it be so restricted. I don’t know, Marianthi, did I get that right? – Yes, actually, thanks for commenting on this. I think one of the side effects, so to speak, if you want, of Sentinel, is to actually push you to think harder about the foundational questions that are surrounding, the very important questions, that they are studying.

And so this comes from the way, and my singling out the, basically, by looking at what the Sentinel has been done, and by looking at how the FDA acts when it takes regulatory actions. And I’m not putting this up, I didn’t put the pyramid, the rendition of the pyramid of evidence in order to say scratch it. No, but in order to single a certain portion of it, where it resides, really towards the bottom. And that is the case series. And ask where it should be put and how can you put it. That we are far away from actually doing that. But this is a question of addressing fundamental foundational issue as to really having definitions. And valid definitions of what adverse events might be. How to find out the, in this sea of data that is being used, local patterns in the data that would correspond to adverse events in the parallels.

If I had to parallelize between the locality in the data of the frequency, that these things, the local patterns appear in the data. And really begin thinking about can we do this automatically? There are very validated, very well-validated and very well-studied case definitions and those can be the fundamental ground that we can begin with to see whether we can push this type of thinking forward. It is possible, and as a matter of fact, it is bringing up the question of how would you do inference based on these local patterns? And an extremely important statistical question, and a statistical question that does have foundations in the field of statistics, but it is quite different of what currently is being done.

I think the Sentinel System provides that particular opportunity with all the things that they have. Not just to use what they have, but actually use this data, this source, to be able to actually create new ways of thinking, and new systems, and new methodological approaches. Which actually brings me into the question of bias, that the two fundamental papers that I actually thought that really get at the heart of the issue in observational studies, is the paper by Schuemie et al, as well as the Gruber and Tchetgen, Tchetgen paper. And their rebuttal of the authors of the first paper.

That is extremely interesting, I do recommend everybody to read, because it does really, it puts forward the Schuemie paper a way of measuring and exhausting for systematic error, rather than trying to eliminate it or ignore it. – Great, thanks, and Joanne, I don’t know if you wanna comment on this, or the effectiveness issue, too, but just a one comment on that. It seems like there is a lot of opportunity here. It does seem like we’re gonna have to go another level further on sort of better, cheaper, faster. This seems like a lot of computational work to do on the systems that are being built. But please go ahead. – Yeah, I wanted to comment on the effectiveness, efficacy in particular. I think that is very important for us to tackle as a community and inevitable. I think we will face that. And so as you know better than anyone, Mark, there are two separate but complementary efforts going on, on by what used to be called IOM, or NASEM, and the other by the Duke-Margolis Center to really try to lay out some principles. – Complementary efforts. – I said complementary, yeah, I said that, to really lay out some principles of guidance in terms of the most important basic elements.

When is randomization necessary, when is observational data good enough and can give meaningful insights into efficacy or effectiveness. But the second element for efficacy is part of this learning healthcare system and predictive modeling. And so if we could look at real-world data to find out which patients have the most benefit from the drug, from an efficacy perspective, which patients are at the highest risk for having some bad outcome, for which there is an approved therapy now. And try to predict that. And we can do that now using a whole host of baseline medical risk factors, but you can imagine in the future, if we’re doing another VTE study, or another AFib study, we’ll not just be looking at baseline risk factors, we’ll be looking at what people buy on Amazon, and what they’re watching on Hulu, and how much they binge watch, et cetera, et cetera.

And how that also factors into the prediction of whether they’re gonna get a VTE, fill in the blank, stroke, et cetera, et cetera. And that’s coming in the future, I think. And all of that really is effectiveness. And that’s a way that we can improve the healthcare system, who we’re targeting for those letters to patients to tell them to, do you wanna talk to your healthcare provider about how your AFib’s being treated, et cetera. And then see whether we’re having an impact on the overall incidence of either adverse events, or efficacy effectiveness parameters.

– Darren? – So I would like to bring a few sort of points that we’re discussing. So I will start with a very pessimistic view, and then I will end with optimism. – Good, I prefer that order. – So we had project in which we talked to real patients, like what do you feel about sharing your information? The patients say, it’s all good, I’m happy to share my data with you, if it will help with improving patient care.

In the end, the patient actually said, actually, it’s not a good idea for you to talk to us, because the reason that you are able to talk to us is because we are open to sharing data. You should be trying to talk to be people who live in the bunker. They are also taking medications, but we have no way to reach out to them. So as an epidemiologist, we always worry about selection bias. Like if we are only able to talk to people that we can talk to, the information will be good, but sometimes it might not be good, because you will be so biased that the information that you are going to generate might not be useful. So that is something that has always been in my mind, I think that having this newer data sources is great.

But the representedness of the patients, and the information, is something that we have to think about. And that gets to the methods that will address this type of selection bias. I think that that is something that we also have to pay attention to. On the more optimistic side, I think that we might just be one generation away from worrying about privacy or security. The reason why I say this of my relatives, so my younger relatives, they could care less about sharing their information. Because the information is being collected in so many ways, and sometimes I just feel like they overshare. But they don’t care, so maybe the optimist side is we sort of move on this in society, maybe information will just be a commodity. That people will be less worried about sharing information. And that would be helping us in getting this information from different data sources to do the work that will be useful for public health. – Any further comments on this or other topics? And similarly, comments from those of you here this afternoon, yes? – Hi, this is Lily Payne representing Booz Allen Hamilton, I’m just to comment on the patient sharing of data.

So if we look at other models like, if you look at Facebook and Twitter, patients, or people, tend to share their data when they feel like they’re a part of a larger community. If you look at Facebook, people tend to overshare their information about their personal lives, so I wonder if FDA could do something to promote that sharing of data among patients with other patients. – So I don’t know if I’m gonna answer question directly, but I’m gonna pull in one of Mark’s earlier questions about blockchain. There was a very interesting article in the New York Times Magazine a few weeks ago. It was nominally about Bitcoin, but it was really about blockchain as a disruptive technology with this point in mind.

But in that author’s mind, the Facebook model was the wrong model. Because it was a centralized model where all the benefits accrued to a single corporation. And in my understanding of that author’s framing of what blockchain offers, one is what all sort of think about, which is this automated trust that allows us to automate a governance model. But the other point that he made was that it creates a token economy, and maybe Mark can weigh in here. – No, you’re doing great. – (laughs) And so I think this is a really interesting question. So that article focused on the patient-level sharing, or the person-level sharing of information. And to make that, enable that in a healthcare setting, it would have to be, patients would have to have their full electronic health data. So we don’t have that system yet, although we’ve been talking about it. But there’s an immediate level, at least in theory. Where the data holders, the data partners, or I would say all the stakeholders in this effort.

So the data holders, the data analysts, the patients, the regulators, whoever else wants to use this data, could participate in a token economy, where if they contribute, whether it be data, or knowledge about methods, or coding, or clinical interpretations, they would get Sentinel tokens that would allow them to participate more fully in this network. So I’ll stop there before I get myself into trouble. But that’s probably like a 20-year vision. – It may be a 20-year vision, but you think about the intermediate steps, and I did mention at the beginning of my comments about the broader changes that are taking place in the healthcare system. Where there is this shift going on towards more patient empowerment and control is coming with these emerging electronic data technologies, and lots of other technological changes, too.

Along with that, with healthcare costs what they are, more and more pressure for accountability that treatments that are being used on patients in real-world practice are really making a difference for them, both in terms of outcomes, and in doing it as efficiently as possible. And I do see, in conjunction with that, a lot more interest. And again, we appreciate any comments from the panel about how to accelerate it, growing pressure, and in a good way, to encourage more use of systems like these to develop evidence on whether a particular combination of treatments, in a particular patient, maybe one with multiple chronic diseases, or one who could benefit from a combination of genomics and all these other dimensions of what could be future computable phenotypes, would really benefit from a particular treatment pattern.

And at what cost? So changes in payment for healthcare providers, for drugs, for devices that are moving in the direction of requiring either more real-world evidence, or actually some demonstration of impact. Subject to certainly some important methodologic concerns there, too. That seems very reinforcing, for what you all are talking about here. And for the steps towards, I’m not sure we’re gonna get to that economy envisioned in that New York Times paper by the Bitcoin, and other coin minters, but certainly in that direction. We are about out of time here. So I would like to, and maybe this is a good, broad question for it, invite any final comments for the panel about how this whole enterprise, certainly FDA led, but by no means, far from FDA alone, how this whole collaborative enterprise can best move forward. Any final thoughts, Bob? – I’ll just make one brief comment. I think as FDA, we always come back to our enabling legislation.

And I think it’s very important that we do that, because we have this, not only do have to do this legally, but it creates the core use case that demands the same, the highest-quality data, the best methods, which then become the basis for framework for these other uses. So I think that’s where, at least my focus is, and I think it, but I think it is really, it does lead do all these other possibilities. – Great, great comments, any others? Well let me ask you stay seated just for one more minute. But for right now, I’d like to thank our panel for bringing together a lot of issues, both long-term vision and, as Bob rightly reminds us, especially in healthcare, it’s those short-term practical steps that really matter. Thank you all very much. (audience clapping) And I would like to thank all of you. Many of you here have made contributions today, both to the panels and to the work that was presented in the panels, as well as comments, questions, making this as rich an interaction as possible. I think that’s very appropriate, given the nature of the Sentinel Initiative and the further work that is going to take place to build on it.

It is very collaborative, and very much requires this kind of ongoing working together to make progress. Though, I do want to thank a few more people specifically, before we end. So obviously, our speakers and panelists, who not only put together great presentations today, but put in some work before this meeting in helping us come up with the agenda. And work out some of the details of the sessions. Special thanks to partners at FDA, in putting this together. Michael Nguyen, Azadeh Shoaibi, Tyler Coyle, Jamila Mwidau, all of you, and many others have worked closely with us over the past months to plan this event.

And last, special thanks to our team at Duke-Margolis, Greg Daniel, Isha Sharma, Sarah Supsiri and Nick Fiore, thank you guys for keeping me on time, too, as we finish up right now. And especially, Adam Aten, for putting this all together. And all of you, have a safe afternoon home. Hopefully the weather’s gotten a little bit better. And we look forward to the next steps on this collaborative enterprise. Thank you very much. (audience clapping) .

As found on Youtube

Comments are closed.