The downtown Denver area is unique. Here there's just so much more to do. There's plenty of historic sites, brand new businesses, museums, and cultural events happening every month. Denver is an extremely active city. It's an excellent place to try just about anything. People have come down here both to live and to play. It's full of shops, restaurants. It's really made it an enjoyable place to hang out. The changes to the downtown area are amazing, from Larimer Square into 16th Street Mall. 16th Street Mall is a really diverse area. Each block is a little bit different and unique. I really enjoy the Denver Pavilions, which has an open-air concept that combines outdoorsy Colorado but also a very urban feel. There's the Hard Rock Cafe, Jazz at Jack's. You can also catch a film at the United Artists Theater. I stopped by the iHeart Denver store in the Denver Pavilions. You can find a real gambit, gift items, fine art pieces, prints, t-shirts, and jewelry, all made by local designers, artists, or creative-minded businesses. Denver has a public art ordinance that allocates one percent of any new capital improvement project to public art. So when you're walking around downtown, you will see a large piece or some sort of art installation in almost every building and plaza. 14th Street in the Convention Center area has been revitalized just recently. The Denver Performing Arts Complex is one of the largest in the country. It takes up four city blocks. You can catch the symphony, the opera, the ballet. It really runs the gamut of popular music to classical. Then you can go a little farther down and hit the MCA. They're just very unique, very innovative with their shows. One of the nicest features about Lodo is the scale. Downtown Denver in general is extremely walkable. If you look at where Careers Field is, even to the Pepsi Center. Lodo is also very bike-friendly. We had one of the first electronic bike share systems. B-cycle stations, you can get on one, you can use it all over town. The heart of the city really is downtown. You have lots of bars, dance clubs, to places with really great views like the View House and nice rooftop decks like the Tavern. Denver's just great for those chef-driven, owned restaurant concepts that you really could pick from about 20 or 30. We have about 92 beers on tap here at Falling Rock. If you're interested in craft beer, Denver is definitely the destination of the United States. Stoic and Genuine is my newest restaurant in Denver Union Station. I love the renovation inside the station itself. It's an incredible transformation of that historic building. They really took care to bring back that era of what it looked like as a train station. So all light rail, buses, and even Amtrak will be coming out of there. It's our pathway to DIA, which takes you to the world. Some of the great places downtown are these historic hotels. One is the Brown Palace. The other is the Oxford Hotel. Also in our neighborhood is the Tattered Cover, which is one of the most amazing bookstores. Just have that really fun boutique feel when you walk in. Just the volume that we have here that makes downtown Denver special. There's really a lot of activity for shopping, too. Larimer Square gives you that kind of feeling like Rockefeller Center might in New York. It's the only historic block that still has all the historic buildings. Larimer Square has some really fun little boutiques that I think you can get some unique gifts. I always take people to Rock Mount Ranch Wear. They're attributed with popularizing the snap front western shirt. From rock stars to ranchers, it's people who are looking for something quintessentially Colorado. If I had one word for downtown Denver, it'd be exciting. Active. Historic but emerging. Be very funky. Be very artsy. It's kind of the heart of this city. It's also a melting pot of our Denver culture. So if you're coming down here, you're getting a real sense of Denver. It's just a constant stream of unexpected surprises here. The downtown Denver area is unique. Here there's just so much more to do. There's plenty of historic sites, brand new businesses, museums, and cultural events happening every month. Denver is an extremely active city. It's an excellent place to try just about anything. People have come down here both to live and to play. It's full of shops, restaurants. It's really made it an enjoyable place to hang out. The changes to the downtown area are amazing. From Larimer Square into 16th Street Mall. 16th Street Mall is a really diverse area. Each block is a little bit different and unique. I really enjoy the Denver Pavilions which has an open air concept that combines outdoorsy Colorado but also a very urban feel. There's the Hard Rock Cafe, Jazz at Jack's. You can also catch a film at the United Artists Theater. I stopped by the iHeart Denver store in the Denver Pavilions. You can find a real gambit. Gift items, fine art pieces, prints, t-shirts, and jewelry. All made by local designers, artists, or creative minded businesses. Denver has a public art ordinance that allocates 1% of any new capital improvement project to public art. So when you're walking around downtown, you will see a large piece or some sort of art installation in almost every building and plaza. 14th Street in the Convention Center area has been revitalized just recently. The Denver Performing Arts Complex is one of the largest in the country. It takes up four city blocks. You can catch the symphony, the opera, the ballet. It really runs the gamut of popular music to classical. Then you can go a little farther down and hit the MCA. They're just very unique, very innovative with their shows. One of the nicest features about Lodo is the scale. Downtown Denver, in general, is extremely walkable. You look at where Careers Field is, even to the Pepsi Center. Lodo is also very bike friendly. We've had one of the first electronic bike share systems. B-cycle stations, you can get on one, you can use it all over town. The heart of the city really is downtown. You have lots of bars, dance clubs, to places with really great views like the View House and nice rooftop decks like the Tavern. Denver's just great for those chef-driven, owned restaurant concepts that you really could pick from about 20 or 30. We have about 92 beers on tap here at Falling Rock. If you're interested in craft beer, Denver is definitely the destination of the United States. Stoic and Genuine is my newest restaurant in Denver Union Station. I love the renovation inside the station itself. It's an incredible transformation of that historic building. They really took care to bring back that era of what it looked like as a train station. So all light rail, buses, and even Amtrak will be coming out of there. It's our pathway to DIA, which takes you to the world. Some of the great places downtown are these historic hotels. One is the Brown Palace, the other is the Oxford Hotel. Also in our neighborhood is the Tattered Cover, which is one of the most amazing bookstores. You just have that really fun boutique feel when you walk in. It's just the volume that we have here that makes downtown Denver special. There's really a lot of activity for shopping, too. Larimer Square gives you that kind of feeling like Rockefeller Center might in New York. It's the only historic block that still has all the historic buildings. Larimer Square has some really fun little boutiques that I think you can get some unique gifts. I always take people to Rockmount Ranchwear. They're attributed with popularizing the snap front western shirt. From rock stars to ranchers, it's people who are looking for something quintessentially Colorado. If I had one word for downtown Denver, it'd be exciting. Active. Historic but emerging. Very funky, be very artsy. It's kind of the heart of this city. It's also a melting pot of our Denver culture, so if you're coming down here, you're getting a real sense of Denver. It's just a constant stream of unexpected surprises here. Well, welcome. Welcome to the session on the lore of big data. Looking under the covers. My name is Helga Rippin, and I'm today's moderator, and the views I express today are those of my own, and they don't represent those of the U.S. Department of Veterans Affairs. As leaders in preventive medicine and public health, we all understand the critical need for actionable data. We understand how biostatistics and epidemiology enables us to better understand what the data means and how to use it, but it's really only one piece. Today we live in a sea of data. New tools are available that can link, overlay, and allow you to explore data in exciting new ways, such as artificial intelligence, machine learning, in addition to new concepts such as data fabric. So during this session, we're going to touch on different data themes that are relevant to all of us as we have faced and are still facing COVID-19. Dr. Joseph Francis will cover learning objective number one, where he will really go into detail and describe VA's pandemic response data ecosystem, and really understanding the challenges that big data could really not solve, and the role of human factors, something that I think we don't actually pay enough attention to. Charles Gabriel will help us understand the capability of advanced and predictive analytics using social determinants of health, to not only really assess the community health, but also how it can mitigate risk and craft intervention strategies through machine learning. And that includes looking at some interesting geospatial tools. And lastly, for objective number three, Dr. Kenneth Berkowitz will touch on really kind of the ethical components, and really describing healthcare ethics concepts related to data access and use, which is becoming increasingly important. I don't know if you've read recently about, I think, Amazon not using facial recognition software. There were some experiments that people were doing in facial recognition in certain cities and jurisdictions that have been halted. And so, when you hear Dr. Berkowitz, you'll be able to consider how you really need to play an active leadership role in ethical practices for data access and use, and really to demonstrate your commitment, but as importantly, your organizational commitment to strong ethics practices for data management. And of course, I am so excited to welcome our speakers, Dr. Joseph Francis, he's the Executive Director for the Office of Analytics and Performance Integration for the Veterans Health Administration, VHA, at the U.S. Department of Veterans Affairs. He's an internist, and he did his fellowship in geriatrics at the University of Pittsburgh, and many of you will love this. He has an MPH in chronic disease epidemiology, and I can just say he has deep, amazing, extensive experience in leveraging data to really improve outcomes. And he led the analytics efforts to support the data needs for the VA to respond to COVID-19. He's joined by Charles Gabriel. He's the IT Project Manager for supporting the Defense Health Agency, DHA, as staff to the Federal Electronic Health Record Modernization Office, lovingly referred to as the FHIRM. And he has served as DHA Chief Science Officer in their Analytics and Evaluation Division, and has worked in the private sector as a Senior Manager of IBM Global Business Services. And lastly, we have Dr. Kenneth Berkowitz, who's an internist specializing in pulmonary and critical care medicine, and he serves as a Special Advisor to VHA's National Center for Ethics in Healthcare. He sees patients, and actually was one of my heroes working to support COVID patients, and supports ethics-related activities in the VA New York Harbor Healthcare System and New York University School of Medicine. He actually has led the development of VA's Ethical Principles for Access and Use of Veteran Data, and is a Chair of the Data Ethics Working Group for VA's Data Governance Council. So an amazing group of speakers, such amazing things to hear about. So Dr. Francis, why don't you start us off? And I can, and see me, and let me see if I can actually work the clicker. Yes. All right. So I'm going to actually run through a case study of what the Department of Veteran Affairs did in the last year and a half in response to COVID in terms of biosurveillance and reporting. But really, this is going to be a case study on the use of big data, some of the things that we could leverage because we have, alongside the Defense Health Agency, probably the nation's largest integrated health record, a lot of information, but also a lot of things that we don't have. And you'll have an understanding of what that is and some of the limits of what we were able to do as a result, but at least some principles that we gained. This is really the second iteration of our national surveillance tool. We call it NST 2.0. And I'll just say as a bottom line up front, one of the things that we found was necessary was first of all, establishing early on in the pandemic, an authoritative source of data. And I will say that in the midst of an exponential rise in cases in a pandemic, no one has right information. It's a bit like measuring the velocity and the barometric pressure during a tornado. You know things are bad, but you will not get a precise number because there's so many things that interrupt your instrumentation. And so we had to settle on a system-wide approach, even though there sometimes were differences of opinion on the numbers, particularly when we were on the steepest part of the epidemic curve. We secondly established a source in order to alleviate burden on our facilities. If you follow the news with COVID, you'll find that the patchwork quilt of public health reporting through states and local agencies, through CDC and HHS and FEMA, created incredible burden on local hospitals and health systems. And one of the things we realized is that if we had to be turning around and responding to that burden of big data requests from all the different players, we couldn't get our primary job done, which was to take care of veterans safely and to take care of our staff. We started this process a year and a half ago, but what we learned over time was that you had to continuously update it. And so while I call this 2.0, I can assure you there was a 1.0 a week later, there was a 1.1, a 1.2, a 1.3. And frankly, I think right now we're probably about 2.7 because every week we find new features that can be added. We get new requests or suggestions from the field, sometimes local sites, develop very clever tools locally that we say, hey, we need to adopt that and move it forward. And so one of the main things we've learned was number one, we needed better breakdown by demographics. The disparities issue with COVID is an important one. We found the disparities were not as great among veterans as they are in the community, but still there were pockets that we had to do a better job. We also needed to do a lot of tracking that would allow local facilities to use this as a clinical management and quality assurance tool. So big data is not just about Big Brother. Big Data to Succeed is about giving local tools to clinicians that can help them understand the course of care. And this is the feature that we called patient story and also patient insights so that we could create visualizations of sort of aggregate movement of patients through the different levels of care, emergency room, ICU, long-term care, but also specific things like are they receiving anticoagulants or antiviral drugs as a way to do, essentially, like a chart review, a peer review, are we following the currently evolving guidelines? We don't have performance metrics like HEDIS measures for COVID because the guidelines are too fluid right now. That will stabilize over time, but we need to be able to have a flexible tool to track the performance. And then lastly, something that I've already mentioned this is not a static tool. We have a lot of smart people in VA. We have 360,000 employees, many of whom query the data system. We have over 30,000 query our NST 2.0, and every one of them have suggestions and ideas how to make this more effective for their work. So let's talk a little bit about what this tool is all about. When we started the design, we realized, when you look at data, big data, there's lots of different views. You need to take a strategic view, which allows Dr. Richard Stone, our acting undersecretary, to see how are we in terms of preparedness and response? Where do we have vacancies? If we see a site that's short on staff, can we send folks from another VA, deploy them cross country, or send supplies or ventilators? All of these things were happening early on in the pandemic. That's the strategic part. On the operational level, if you have an emergency, a national pandemic emergency, you have an emergency command center, you need to have focused tools site by site so you can see where the hotspots are, and those tools have to be available to local emergency command centers. We had a tiered response. So there was an EMCC at the national level that met every day, at the regional or VISN level, the Veterans Integrated Service Networks, and at the local facility level. And then the last part, which is the tactical one, is the day-to-day management and the tools and what facilities did on a daily basis to manage the care of COVID patients. All of these views were built off of a single data mark, which was our NST data mark. We developed that in partnership with our Office of Information and Technology and our Office of Public Health, and many others. And that became the basis, not just of these three views, but also everything that we were exporting to other agencies. And we synchronized this tool to our public reporting. So we didn't have to worry about updates or about data conflicts. Everything was platformed in the same data set, synchronized at the same time. And if you go to accesstocare, one word, .va.gov, you will see today exactly how many patients were taken care of with COVID, how many we've had to date, how many have recovered, how many have died. You'll also get a breakdown of vaccination status, and you'll be able to see that breakdown. We just released an update of vaccination status with demographic stratification. So you can see gender, race, ethnicity, and all of that. And I recommend that you maybe do that during the break on this conference. By the way, this data mark had four basic flows of information. The patient data came from our electronic health record. We have separate information about our employees. So that was staffing information. We have, I'll say we were in the middle of implementing a national logistics solution. So probably the weakest link here was our inventory on supplies and equipment because that had not been fully implemented. And then our capacity data refers to bed data. We have a bed management system that's also on a national level. And that has been a system that's been continually updated, again, in flexibility for the response needs of this pandemic. So again, a little bit of how we got this information. If you look at the patient data, here was one of the dilemmas. We didn't do all the testing for veterans known to be COVID positive. About 60% of the testing happened within a VA hospital. That's the easy part. How do you find out when somebody has been tested in the community when there isn't robust real-time health information exchange? This was not happening. It still isn't happening for the most part. What we were finding out was the best way to get it was when a veteran came in and told us. So we had to figure out a way to query text notes by doctors and nurses with the information that they typed in about the veteran status of testing in the community. And that's about 30 to 40% of the cases that we identified. And by the way, we had a continuous process of chart review. I put my clinical team on a continuous review. The querying of the progress notes was being done through machine learning and natural language programming. But because how we tested was changing so rapidly, we had to constantly retrain the system. And so yes, AI and NLP can help you, but we had tens of thousands of hours of our own staff reviewing cases and confirming either the positive or the negative state. In some cases, we were doing 100% review, and this was taking us gobs and gobs of time, amazing overtime. Don't think for a minute that machine can do this for you. And by the way, we're doing this not just for veterans, but we're doing this for staff. Some of our staff are veterans, by the way, and the staff that are not veterans, we get that through our other workforce systems. And then we had other cases show up. We've been caring for the community. We've had mission, what we call fourth mission assignments. The fourth mission of VA is supporting the nation's national emergency response. So we had to find ways to pull that information in, flag those patients so we didn't confuse them with veterans and make sure that that information stayed segregated from our rollup on veterans, but was still transparent to the users of the system. This is a little bit about the cadence and the underlying data feeds. And you'll notice here that some of the things get updated on a daily basis, but most of what we were doing was actually being updated on an hourly basis, whether that was from bed management or about patients. Frankly, big data that's a daylight in a pandemic is useless. So it is really, really key to keep up the cadence. And this became important so that we had to ask, again, our colleagues in the Office of Information and Technology to devote extra resources to ensure that the system was up and running. We had a few periods of downtime, and I can tell you sometimes the downtime was the result of complex system failures, things that we had never anticipated happening because the system never had a load this great on it. This was real-time learning that all of us learned a bunch of experience from. This is what the system looks like. This is what it looks like if you're on the strategic level. This is what Dr. Stone sees every day. He takes a printout of this to the meetings with the COVID-19 task force that the White House chairs. And we have a map there that emulates the Hopkins map, if you've ever checked on that. You can see the total trend lines for cases, for deaths, for the different categories that we took care of. At the time were all those other patients, they weren't our employees, but they were other folks, might've been humanitarian. And again, breakdowns in views by demographics, age, days hospitalized, and many other factors. Just to blow up one of these to give you a sense of the drill down, we're able to see in real time the age breakdown and race, ethnicity. This was very, very important for us. As you know, COVID was a condition that generally hit ports of entry. Our big cities, our big VAs were hit hard, hit early, the same ones as in the private sector. So our New York Harbor and Bronx facilities, Boston, New Orleans, San Francisco, Seattle, Denver, Detroit, these were all sites that got hit really hard. And these are also the sites that had a higher proportion of minorities. Gonna go back a little bit here. I think I went too fast on the clicker. So now let's get down to a little bit more granularity. What you're seeing here is also trending. I wanna point out something here. We have a toggle feature because you can't put all this data on a single page in a dashboard. So you need the flexibility with big data to allow for a quick jump to different views and not have to go to a different website or a different reporting system. So this allowed us to maintain some parsimony. It's an important user feature, but also the flexibility for folks that might wanna look at things in a different way. And here's something that we thought was very important to why. This is our excess mortality graph, our trend line for deaths to date in the prior fiscal year and the current fiscal year. And you see where the lines diverge, that's the excess mortality. And I'll just say that the excess mortality turned out to be higher than the known COVID deaths. Not surprising, given that there were deaths in the community that we may not have been notified in real time about. There was often a delay in processing for burials or getting death certificates through. And there may have been deaths because of deferred care. Without an autopsy or other clinical information, it is very hard to understand all the factors going in. But this is no different than the private sector and the reports that you're seeing from CDC and other groups. Again, a little bit of a blow up on the excess mortality charts. And by the way, you can make these charts fancier. We have another system that we don't use for daily reporting because it gets to be overwhelming. But there are some Bayesian techniques to anticipate what might be a delay in a death notification. You can adjust your excess mortality tracking with those techniques. But that ends up becoming very hard for a manager to use. Just try to explain Bayes theory to an overwhelmed medical administrator at a hospital. So we use that more to get a firmer estimate of system burden. But that's not something that we put into the reporting tool. Again, because of respect for what our system leaders and managers need to be looking at. Other features, we were very concerned about turnaround time for testing. So we tracked that a lot. Again, we have strategic displays for this. Now we're gonna get down into some of those finer grain details, which refer to the patient's story. And this again, gets down to the facility level. And you can see again, the ability to track the kinds of interventions that were done. And we didn't talk about this, but within this actual report, a user that is authorized to get individual record level data, we call it SSN level authorization, can click and get down to the exact patients. And then you can link to the electronic health record and do your own review on the fly. Again, just a few other screenshots. I realize I'm probably going over time here with the various levels of detail. So I just wanted to give you a sense all within a single report, a single website, the richness of the information that we were able to provide. And I wanna leave just with this final piece, when we built the tool. And by the way, the infrastructure for this, a lot was constructed before COVID hit. Our biosurveillance team was very worried about pandemic flu. We knew we had to have something like this in place. You don't build a system like this overnight or in the middle of a category five tornado. So we also knew that we had to be working with others. So when we built the tool, we built in the interfaces, whether it's with other types of reporting systems, like we call this symphony. It's one of our high level GIS enabled tracking systems in VA. We have a partnership with VA research. So we send a copy of the data set to our research computing environment. And you have seen a number of very important papers come from VA just in the last two weeks, even a paper in nature, looking at the prevalence of persistent COVID symptomatology. We've mentioned the White House, FEMA, HHS, CDC, in multiple CDC platforms, NHSN, as well as other platforms. And most recently, we've actually been able to find a way to ship information to individual states. We've developed a partnership with the Association of Public Health Laboratories. And this allows the states to have window into what's happening with veterans. We're working now for the next phase of this bi-directional exchange, so that states can get information to us directly about veterans that may have been vaccinated or may have been tested in state sponsored events. That's something, again, that we are currently not getting, except when a veteran comes and tells us about it and we capture it in a note, which then our system can capture. And I think that takes me through my presentation. So back to Helga and your next speaker. Should I start on? So happy to be here with the very elite panelists and a special honor to be presenting to the American College of Preventive Medicine. And I just wanna start off with what Dr. Francis emphasized that this crisis is not only a health crisis, but he touched on the information part of the crisis as well. And I totally agree. Hopefully, this crisis will teach us positive that can come out of this crisis. And I think we need to be prepared for, God forbid, next health crisis or national crisis. So I wanna start with the definition. So these are the two common definitions for determinants of health. And I'm using the term determinants of health to be comprehensive, because it's not only a social. And as you can see in the second one, not the World Health Organization definition, but the second one, which is from the Office of Disease Prevention and Health Promotion, I'm highlighting the personal, social, economic, and environmental. And there is a reason to separate those type of factors from a data and measure perspective, but happy to answer any questions about those as we proceed later in the presentation. But what I wanna emphasize also, aside from definition, that there is a very strong, as I'm looking at the data and I worked with distance, I worked on determinants of health since 2016, there is a strong correlation between the preventive medicine and determinants of health. And I'm happy to hear any statement or a challenging question about this statement, but definitely there is a strong correlation between the preventive medicine and determinants of health. Next slide. So this is kind of gonna, I'm gonna go over the kind of the study that we did to use determinants of health to understand the National Guard population, wellness and readiness. And the study was focused on seven states, Florida, Indiana, Maryland, Oklahoma, Minnesota, Mississippi, New Mexico, and about 461 counties. And we focused on many health factors, but seven was kind of a primary focus on the study. So the health factors were obesity, excessive drinking, tobacco, economic indicators, hypertension, access to care and substance abuse. I'm not gonna present everything, but I'm gonna touch on some of the stuff and how we tackle this population, determinants of health. So the next one, here are the high-level goals or questions that the leadership wanted to understand on the DOD level and the National Guard specifically. So the first one is, can we understand the problem space? What exactly the common set of prevalence in those areas or in these geographic areas? So that's one big question. And if we do the assessment as we did in the study, really the assessment, even the initial assessment, kind of may lead to the root cause of the problem, which is a big deal. The second piece is evaluating the priority. Try to understand the magnitude of each problem and the severity of the problem on the society or county or state level. And the third one, OK, if we address those health problems within the community, how do we shape the strategy? What can we do to do strategy based on the geographic area? Or there is something else we can do to shape a strategy that's meaningful and improve the health and wellness of those community. Moving on. So this, I'm just showing you the visual piece of the states are involved in the study. And you can see the population of National Guard of each state. And this by itself is also another factor to evaluate the significance of a problem, assessing the problem, and understanding what really a meaningful intervention, a candidate intervening intervention. Give it a few seconds for you to get a sense of the areas and the map and several population. I'm going to move on to the next slide, if that's OK. So this is kind of video to show you. As you press on those bubbles or on those counties and change the parameter, say I want the population over 600 or 1,000, whatever that is, those bubble represents the National Guard within those communities. And you see the shaded area, blue, yellow, and greenish type of shaded area. That's show you the per capita income. So of course, it shows in the visualized kind of context the problems, the within the problem, and so on. And there are some kind of two key parameters to not to make it so noisy for the users. But as you switch parameters, you definitely can change those. But that gives you kind of an essence of how you relate the problem to the environmental or the social factor. Next slide. So this is the machine learning piece. I remember I emphasized that, OK, how do you intervene in the community? And you have seven states, and you have 400-plus counties. And definitely, if you want to shape the strategy, the best way is not to shape it based on the geographic area, but also based on the strengths and weaknesses of those communities. So this machine learning clustering visualization that you see does exactly that. It groups the community based on their social determinants of health or, in general, problems and strength. This is kind of the last piece, which is the methodology that's used to come with this kind of the analysis. So behind the scene, behind those visualized kind of algorithm that you see in a nice user interface, there was integration of two big data sets, the public data sets, state and county-level data sets, EHR data sets from DOD and a population of the National Guard. There was also kind of enrichment on those two data together. There was measuring of those common factors that we talked about, the determinants of health factors, whether it's tobacco, obesity, drinking, and so forth and so on. And there is the scoring behind the scene, the modeling, and the verification of the modeling, the discovering piece, and then, finally, the visualization that you have seen. I think I hope I was not too fast, but I'm allowing more time for my next speaker, but also for more questions. Over to you. Hi, everyone. Thank you, Charles, Joe, for those great presentations. And I'm going to be talking a little bit today about how we do all this stuff within an ethics framework. Because when you think of health care quality, think of technical quality that we have to meet medical standards for the health care we provide. You think of service quality, things like access, satisfaction of our patients and our staff, and equity. And the third leg of that stool is ethics quality and how we operate as health care systems, whether it's in delivering health care or operationalizing the way we use our data, is critical to saying that we have a high quality system. So like Dr. Ripon, I want to just remind everyone that these are my own views. But I also want to say how much I've been looking forward to speaking on this panel, not only because of my colleagues, because of you in the audience. As Dr. Ripon said, you guys, you folks, are often in a leadership role when it comes to ethics. And I'd love to hear what you think about what we're doing and our approach, and to get you thinking about ethics as an important aspect of your data management strategies. I don't think that I need to emphasize how important it is for our organizations to be learning organizations in the truest sense of the word. We are absolutely duty bound, ethically bound, mission bound to promote innovation, to assure quality, efficiency, accountability. We must be learning institutions. However, we also have to realize that we However, we also have to realize that the more we use data in new and innovative ways, the more we increase access to data, the more we erode certain boundaries, the more we mix the clinical care with quality improvement, research, surveillance, oversight, and other things. The more we risk erosion of privacy, the easier it is to slip up in terms of equity and discrimination. And worst case, the more likely it is for us to erode trust in our systems. It's really important. And VA is operationalizing data management strategies that continually strive to promote strong ethics practices. And this is an important organizational responsibility that we've accepted. We recognize that we need to balance protection with overprotection. We don't want to create barriers to appropriate data use, but we want to assure that things are appropriate. And we, in particular, must always include veterans' perspective in what we do and make sure that there are no surprises. We're very fortunate to have very strong leadership support throughout VA from the top of our organization, from the Secretary for Veterans Affairs, all the way down. I put up this quote, which I really try and keep in mind for everything that we do, from Mr. Sanaka, our Director of Information Security Operations and our Chief Privacy Officer at the agency level. And I'm going to read it. Behind every piece of information or data we secure is a human being who has a right to the protection and privacy of their data. And people really do care what happens to their data. Sometimes they're resigned to the fact that things happen that they don't like. But again, this is our responsibility, and I think your responsibility as organizational leaders. And some people say, Ken, why do we need to think about ethics? We have laws, we have rules, and everything will be covered as long as we stay within the rules. And I couldn't disagree more. The rules are important. The laws are important. They develop, set one kind of a framework. But at the same time, the rules-based approach has limited impact. So we need to go beyond that. Consider HIPAA for a minute. It's from 1996, when there was pretty much no big data, no internet. And you could see from this illustration of an iceberg, which came from a review article in Nature Medicine about data and data ethics, that it's a tiny fraction of health information that is even covered by HIPAA. And it's generated by HIPAA-covered entities. And the bigger part of the data iceberg is below the surface. It's not easy to see if you don't think about it. And just as an example, the second layer, which probably should be bigger, is health information controlled by entities not covered by HIPAA. And just consider within the past few weeks that Blackstone Financial, which is a financial group, huge, not a healthcare entity, bought the Ancestry.com database for $4.7 billion. Now, that's people's names, their information, their demographics, and their genomes, okay? And the right to do anything they want with it. It makes GlaxoSmithKline, I believe, purchase of Ancestry23.com's database a few years ago for 300 million seem like a bargain. Think about the fact that also within the past three weeks, Microsoft bought Nuance Voice Recognition for $17 billion. Now, Microsoft, obviously not a HIPAA-covered entity, is now gonna be a leader in voice recognition. And when you think about what tech companies are doing, what big companies are doing, and who has access to data, you can understand that the rules didn't ever even come close to envisioning these types of scenarios and really don't provide the safety that we think. Now, some people say, well, let's fix the rules. And to me, that's like the tax code, is anytime you fix it, you're gonna have a whole bunch of people looking for loopholes. So what we've been advocating for is in addition to those rules, laws, policies, which are critically important, is to develop an overarching organizational ethics framework to think about data. This demonstrates our commitment to our stakeholders and to our staff and our patients. It strategically differentiates us from other organizations and shows our commitment to ethical data use. By having clear practices and processes, we promote consistent and sound decision-making. And by talking about all these things, we advance discourse. When we put this in place, again, as I said, we have internal consistency. It's very practical. It's broadly applicable. It's nimble. And it acknowledges that there are trade-offs and balances that we have to think about if we're gonna justify what we do. Now, these frameworks have different components. There's values and mission. There's principles that undergird everything. And then those lead to rules and guidance and structures. So in the Veterans Administration, we have clear values already. We have a very strong mission. We have system-wide ICARE values, which have been delineated. We developed over the past few years sets of principles, data-guiding principles, ethical principles for data access and use, which Dr. Rippin referred to and I'm gonna talk a little bit more about. We have privacy principles, customer experience principles. And those all underlay our rules and guidance and structures which is really how these things get implemented. We have a data strategy which pays explicit attention to ethics. We have a data management directive which codifies a policy that codifies some of these things and commits us to following our principles and our processes and we have or are passing regulations to make this backed by law and give us the legal authority to do this. We have strong structures because if another way to think about ethics quality is structures, processes and outcomes. And if you set up the structures and the processes that leads to at least you know how you got where you're going. Ethics is part of the data governance council. There's a data ethics work group, a standing work group of that and a data ethics board to make a high level precedent setting decisions and there's a privacy steering committee. So we're weaving this into the fabric of our organization on paper and in practice. Now to illustrate some of this, I think of information as traveling along kind of a highway that's sort of our metaphor these days for it. And there's different lanes. So way on the left sort of in the fast lane when you're considering data use, there's things that we must do as an organization. We're required by law to make certain reports. We have to report to Congress. There's other things that the organization really must do and that's sort of the fast lane. We know we should do it but that doesn't mean that how we do it isn't still important. So even if you're in the left lane, the straight lane to that we should do this with our data, we still have to pay attention to how we do it, our values, our principles and make sure that what we're doing is transparent. There's no surprises. It's consistent. It's fair. It's equitable as we move towards should. On the other end of the highway, there's things we can't do that are illegal or that would just be just flat out wrong or not consistent with our policies. And those things need to be recognized sort of early on and you need to exit. Say you're gonna come to a dead end or I'm gonna have to change the rules or the laws or the policies to make that sort of inbounds. And most things though, fall into that middle lane of proposed data uses that we can do. And that's where we have to go a little bit slower. We have to have those to pay attention to our deliberate processes and our deliberate and consistent thinking so that we can justify whether we should proceed and how we should proceed or whether we should take an exit and stop and say, no, this really isn't for our organization. So I think that the prior presenters presented some very high level, big, important data structures, strategies and uses. Although Dr. Francis pointed out that with the right authority, you could drill down to the patient level data and get down to it. So who has access to that, what they're allowed to do, how they're accountable for it. Very important questions. But there's countless other questions. So consider the example of a third party app and this is more and more common that wants to provide patient access to their electronic health record. And they've come to us and they've requested access to the health records through the application program interface, the API. And the question is, should that be granted or not? And one of the things that we've put in place that I sort of alluded to is principles for access and use of veteran data. So sometimes when you get down to these, considering these granular requests, it's important to think about those roadsides along the road. And one of them is, would this access be consistent with our standards? And again, these standards were approved by the secretary of VA, their informal VA policies, and they're the fabric of our organization. And those principles for access and use of veteran data, whether it's for clinical care, education, quality improvement, research, or other data analytics, is the first thing is that the use has to be for the good of veterans. The next thing is it has to promote equity. It can't promote inequity and we have to really be on the lookout all the time to make sure that we're inbounds when it comes to equity. Third, to the extent possible, veterans have to be given meaningful choice about whether it's something that they're okay with. And if it's something we're gonna do and they don't have choice, then principle four, transparency, really comes into play. Again, there shouldn't be surprises amongst patients, staff, or stakeholders. Five is principle de-identification. And really, when it comes to big data these days, it's very complicated to think what is de-identified data and it's arguable that nothing is because almost everything can be re-identified. So rather than try and make it so you can't be re-identified, we also want people to commit that if we're giving them access to something that is de-identified, they're not gonna try to re-identify it. They're not gonna go beyond the scope of what they were authorized to do. There's a reciprocal obligation to veterans to give back, to recognize the contribution of veterans and to include them in thinking about what stake they have in the game that comes from the use of data. There's an obvious obligation to ensure data security, quality, and integrity, and make sure that we have, that that's maintained as people use the data. And last two, sort of real common sense, is veterans, this is their information. They have the right to access it and they have the right to request an amendment if they feel that there's an error or something needs to be corrected. My clicker is not working. Let me try again here. Technical assistance, please. I am lost in the presentation. This is not our presentation. Okay, let's see if I can get us back where we were. Trying to go one more slide forward to the back to the example. My internet clicker, two more slides forward. I'm sorry, folks, I can't get to the next slide. There should be more slides. Anyway, I'll try and talk through the slides. I don't know what happened to the end, but we were considering that, and I'm almost done, that application of the principles to the third party apps request for API access. And when the group came to us and said we have concerns, we walked through as one of the steps going through their request against the principles. And when we talked to it, they said, well, this isn't clearly beneficial. There's better ways for our patients to get access to their records. And more important than that, there was inequities felt to be in the way that the app was gonna work. There was no sense that it was secure, that the contractors, the vendors, wouldn't commit to holding their subcontractors to our same standards. The app signup was a boilerplate sort of user agreement that was no meaningful choice. There wasn't a lot of transparency in what else the app developer would do with the data once they got it. And when they looked further, their intention was to monetize the data, which is buried, not buried, included in one of our principles that goes against the obligation of reciprocity and transparency. So it was felt that the request would be denied, that it wasn't applicable, consistent with our principles. And since it was sort of in that middle lane on the highway of the discretionary use, they decided that it was not a justifiable use of the data. So in summary, data has a tremendous potential when it comes with great responsibility at the personal and the organizational level. We, again, I wanna reiterate, this is not to be obstructionist. VA has an unwavering commitment to innovation and being a learning organization. But how we operationalize our data management really matters. The ends doesn't automatically justify the means. And data access and use has to be actively managed to balance protection and overprotection. And it must be ethically managed so that we know all along the way we're promoting strong and justifiable practices in a transparent way with accountability so that we can maintain the trust of those that we serve, our patients, our staff, and our stakeholders. Well, thank you, I look forward to discussion. And so, do you want to go to the question session? Well, thank you all for great, great talks. And now we have the opportunity to answer some questions. So, let's start off with the first one. How has the transition to a newer EHR like Cerner affected the VADOD big data? And Joe, I guess I'll pick on you first. Well, that is still a nut we're trying to crack. And I'll say that Cerner has a lot of very favorable features that we're looking forward to using. But as you've probably heard, we are in the midst of a strategic pause to make sure that the rollout can be conducted safely. It's challenging to put in a new electronic health record, even in the best of circumstances. And the last year has not been anywhere near the best of circumstances. So, there are many factors, user factors related to the health record, as well as to the context of its implementation that are being looked at very carefully. I am very optimistic in the long run that Cerner will provide net benefit to DOD, NDA. And we are working with our colleagues in the Office of Electronic Health Record Modernization to make sure that we have some of the features like interoperability and public health reporting requirements built into those processes. Thank you, Joe. And Charles, given that you're, since you're a part of FIRM, why don't you go next? So, I'm aware that the FIRM is doing serious, serious good stuff in the data management plan. The DOD, definitely DOD and VA are convening to lay out the playground for big, big data plan and data strategy. So, it's a work in progress, and definitely there's going to be some good coming up, hopefully, soon. So, I will leave it to the FIRM to answer your questions. And I guess I'll add something real quick. I do know that the chief data officers for both VA and DOD are collaborating to talk about kind of not just the EHRM data or the FIRM kind of component, but the bigger kind of opportunities. And I know, Dr. Berkowitz, you probably have something to say on the initial, on kind of ensuring a common frame. Yes. And one thing that I'm very happy about is that in the Cerner development, there was 18 councils that were developed to assure VA SME input into the development. We, early on, pointed out that we need a, we would I think it was a 19th council. There's formed an ethics council that worked closely with our DOD counterparts. We have three main work streams. One is to make sure the new record supports our interoperable policies in obvious ethics areas like informed consent, life-standing treatment policy, disclosure of adverse events, et cetera. The next is to provide a consultative resource if people in the electronic health records modernization process have ethics questions. And the third work stream is, relates to data practices and data access and use. So we've been at the table throughout. And I appreciate that. And I'd like to think it makes the system stronger. Okay. The second, thank you. The second question is what penalties, if any, are imposed on those who inappropriately re-identify data? And so I will defer that to you, Dr. Berkowitz. Well, that is still a work in progress. What we're trying to do is to embed the language into our contracts, memorandum of understandings, business associate agreements, research approvals, et cetera. So at a minimum, I would think that they have the penalties, if you will, of violating any of those agreements. But probably most importantly is you'll lose your access, I hope. And you'll be sort of like at the register in the supermarket when we used to go to supermarkets, they used to have registers. You'll be on your bed. The bed checks will be up there. But so it's to be determined. But the goal is to develop real accountability for this. And we're working out the details. Yep. And I, again, I would just hope that we would also take them to court. So also, you know, penalties. Go ahead. Sure. Yeah, I was just going to say, we struggled with this when we were deploying NST 2.0 because it also included a feature to determine the COVID status of our employees that were tested in VA. And so we got pushback from folks saying, oh, my God, how are you allowing to do that? And, you know, this is one of those things where it's a pandemic. If you throw up too many bureaucratic hurdles to the access of data, then the people that need to look at it is going to have, are going to have issues. And, you know, we want this to be very real time. There are approvals to go through. This is no different than accessing the electronic health record in any hospital where, you know, we do have audit controls. You know, if you're taking a look at things and, you know, we see something we don't like, you get more than just a point on the shoulder. We put a disclaimer screen reminding people about what they learn every year with privacy and information security training, which is, it's a need to know. And again, we had the provision, although everybody in VA can see active data, only those that have gotten specific approval and signed off by their supervisor can actually get the individual record level data. So, you know, we also think there's some role of the facility and supervisors to ensure that their staff are following rules and regs properly. And being transparent about the thought that went into those complicated decisions so that there are no surprises and promoting that discourse, very important along the way. Okay. We have a few more questions. The expanded use of consumer genetic testing and now the sharing of non-HIPAA entities and even potentially China through acquisition has prompted DoD to review through the Defense Health Board this concerning trend among military members. Does the VA have any such effort or initiatives in this regard? Anybody from the VA side? Do you know? I don't know. And at the same time, I think the importance is to, the important thing is for consumers to be aware of exactly what they're signing up with and to be aware that those consumer things, those sort of non-health uses, could be married to the health data and can really expose a lot. So I think people need to be educated and go very cautiously in those deep waters. Yeah. Well, there was one more question, but I know we're out of time. So if anybody knows, is there any more specific population health tools that are popular and effective? Any last word, Joe, on any tools that you want to shout out before we end? Anything new, Charles? Yeah, we could have an entire session on population health and what that means. You know, we didn't have a chance to talk at it, but my office has a real-time electronic quality measure platform, which is really looking at your basic chronic disease and prevention management metrics, the HEDIS measure set, so to speak. And we've been tracking that very closely because with the fall-off in face-to-face visits and the move to virtual, there are some things that you can do on virtual, but others you need a face-to-face visit for. Yeah, well, and I— It's a different performance, and we are tracking that very closely, and we're seeing the upturn, but we obviously want to make sure that we can get back to normal operations as soon as possible. Yep. Okay, well, and I know we're way over, so again, I want to thank all of you who have taken the time and attended our session and those that have done such a fabulous job answering questions and giving insightful presentations. And I wish all of you a fabulous, fabulous rest of the day. Take care. Bye-bye.

video

Dr. Joseph Francis

big data

pandemic response

data tracking tool

determinants of health

data visualization

ethical considerations

organizational ethics framework

transparency

electronic health records