Project Name:

Fraud in Public Healthcare Programs: Developing Fraud Detection Models with Linked Data

Contractor: CORMAC Corporation

Lessons Learned

  1. Access to federally protected data is complex and time-consuming.

a) CMS data environments allow authorized users to access CMS datasets under datasharing agreements and to perform analyses across those datasets. However, gaining access is a multi-step process that can take several weeks, beginning with the approval of a data use agreement. Additional requirements include completing business and technical training (e.g., database and BI tools), configuring VPN access on local computers, and navigating agency-specific procedures. Timely progress depends on support from internal CMS resources and clear, open communication channels to resolve issues efficiently.

b) Early decisions about the data access and analysis environment are critical. Any environment used to access federal data must comply with federal cybersecurity and data security requirements. The team learned that access to a secure agency environment could have been expedited if an interagency agreement had been established before the project launch.

  1. A narrow analytical focus is essential for short-duration projects.

Given the project’s limited timeframe, it was critical to work closely with the Department of Justice’s Fraud team to focus on one or two related types of fraud that are both high-risk and feasible within existing data and system constraints. The following factors guided the selection of the project’s focus:

  • Data availability, accessibility, and usefulness
  • Likelihood of fraud based on known or similar cases
  • Care setting
  • Billing or business practices
  • Provider type
  • Geographic location
  1. Careful definition of key variables is foundational to AI/ML-based fraud detection.

During the first quarter, Cormac focused on identifying the data elements and contextual information most likely to indicate fraudulent activity, as well as the analytical processes and tools needed to flag suspicious patterns. The team also considered the required outputs and how results should be communicated clearly, including in plain language, to support effective interpretation and use.

The project had the following lessons learned for this quarter that could inform a future NSDS: 

  • Structured sprint execution strengthens delivery clarity and accountability. With a short performance period of work, we decided to adopt a defined sprint framework to enable a clear sequencing of technical work, resulting in improved visibility across complex, interdependent tasks.
  • Foundational data engineering demands rigorous upfront validation. As we began exploring the data, we needed to build a master list of providers, episodes, and claims structures, which required iterative validation steps to ensure consistency across datasets, underscoring a future NSDS would need to allocate sufficient time for data preparation.  
  • Metric development is most effective with iterative testing. In addition to setting up rigorous validations at the start, we needed to break metrics into discrete, testable units, which allowed us to improve accuracy and transparency while enabling early detection of data quality issues.  
  • Data access timing remains a critical dependency.  Because we continued to experience delays in securing access to the compute environment, we resequenced the work, highlighting the importance of contingency planning in data-dependent efforts.  
  • Continuous stakeholder engagement ensures technical alignment. For the success of any project in an NSDS, ongoing collaborations and discussions with subject-matter experts and clients (i.e., DOJ and NSF partners) has helped keep metric development and analytic approaches relevant and aligned with project objectives.
  • Data Integration is a Critical and Resource-Intensive Phase Linking claims, enrollment, beneficiary data and clinical surveys is typically the most time-consuming and complex stage, requiring significant effort in data cleaning, standardization, and entity resolution.
  • Attribution Requires Advanced Modeling Due to Data Limitations Data used to identify referring and certifying providers is often incomplete or inconsistent. As a result, teams should be prepared to address these gaps by developing robust attribution models to accurately infer provider relationships. 
  • Data integration requires multiple iterations, validation cycles, and refinement of linkage approaches. Data quality issues and incomplete identifiers may require iterative refinement of linkage strategies and attribution methods to improve coverage, accuracy, and reproducibility to establish a reliable Master Provider Layer and a robust analytical dataset for feature engineering and downstream machine learning analyses.
  • Administrative data require robust entity resolution. Although standardized identifiers such as NPI, CCN, PAC_ID, and BENE_ID enabled high deterministic match rates, provider organizations frequently appeared under multiple administrative records due to ownership changes, enrollment updates, and historical billing relationships. Combining deterministic linkage with standardized entity resolution substantially improved provider consolidation and analytical consistency.
  • Referral attribution requires alternative methodologies. If referral relationships cannot be reliably determined from a single claims field, alternative attribution methods may be needed to identify referral providers, certifying physicians, and other administrative relationships to construct meaningful referral concentration and provider network indicators.
  • Peer cohort benchmarking improves statistical comparability. Normalizing provider behavior within peer cohorts defined by geography, provider size, ownership type, and beneficiary severity substantially reduced structural variation that would otherwise bias national comparisons. Hierarchical cohort assignment and fallback rules allowed nearly all providers to be assigned to statistically meaningful benchmarking populations while maintaining sufficient cohort sizes.
  • Feature engineering benefits from multiple risk domains. Using a multidimensional approach that combined operational, financial, referral, organizational, and clinical indicators into a comprehensive feature set better captured the complexity of provider behavior and supported more robust anomaly detection than any single metric alone.
  • Unsupervised learning is well-suited for program integrity analytics. Confirmed fraud labels are limited and often incomplete, but unsupervised anomaly detection methods proved appropriate for identifying statistically unusual provider behavior. An ensemble model produced a more stable composite anomaly score than models used individually.
  • Explainable AI improves analytical transparency. A surrogate model combined with explainability provided interpretable insights into the factors contributing to elevated anomaly scores. Rather than functioning as a “black box,” this model identified which engineered metrics most influenced anomaly detection, supporting analyst interpretation and communication with non-technical stakeholders.
  • Cohort-level aggregation supports both privacy and interpretation. Transitioning from provider-level outputs to cohort-level summaries preserved much of the analytical value of the machine learning models while supporting Statistical Disclosure Control (SDC) and CIPSEA requirements. Aggregated anomaly indicators, cohort benchmarking, and regional hotspot analyses offer actionable statistical insights without exposing information about individual providers.
  • Geographic hotspot analysis is more informative than individual rankings. Aggregating machine learning outputs to geographic and peer-group levels shifts the analytical emphasis from identifying individual organizations/providers to identifying statistically unusual patterns across comparable populations. This approach is better aligned with population-level monitoring, resource prioritization, and disclosure controlled dissemination.
  • Selection of anomaly detection algorithms should consider both detection performance and scalability. Selecting complementary algorithms that balance detection capability, computational efficiency, and robustness provides a more scalable and reliable anomaly detection framework. Several algorithms assessed could identify unusual provider behavior, but neighborhood-based and density-based methods showed reduced effectiveness within higher-dimensional feature spaces and greater sensitivity to parameter selection.
  • Scalable explainability requires balancing transparency with computational efficiency. Explainability approaches should be evaluated for their ability to scale in an operational environment. When an approach is computationally impractical, surrogate modeling approaches can be effectively used to approximate the anomaly detection framework and generate feature-level explanations.
  • Simplicity improves interpretability. Business users consistently preferred dashboards that emphasized high-level risk summaries, geographic comparisons, and concise explanations over highly detailed metric displays. Default views comparing national benchmarks to selected geographic regions were easier to interpret than complex multi-series visualizations.
  • Design around user questions rather than available data. The most effective visualizations are those designed to answer specific business questions instead of displaying every available metric. This approach also helps to prioritize the information presented and prevent unnecessary dashboard complexity. Individual statistics were more meaningful when accompanied by benchmark comparisons, historical trends, or geographic context. Provide reference points to help users distinguish between normal variation and findings that warrant further investigation.
  • Explainability should be presented as risk drivers. Presenting the leading factors contributing to elevated cohort risk proved more meaningful than displaying individual model outputs. Summarizing the top contributing risk domains provides clearer analytical context for investigators, analysts, and policy reviewers.
  • Evaluate visualization technologies early. Understand visualization requirements early in the project to ensure that proposed solutions can provide the desired functionality and performance. Additional libraries and implementation approaches may be needed to achieve the desired functionality and performance.
  • Disclosure controls should be incorporated early in dashboard development. Designing the dashboard with disclosure protection in mind simplified the transition to a CIPSEA-compliant statistical product. Suppression rules, cohort aggregation, geographic rollups, and restricted drill-down capabilities were more effective when incorporated during design rather than retrofitted after implementation.
  • Statistical disclosure control extends beyond data suppression. Producing a CIPSEA-compliant analytical product required more than removing identifiers. The project demonstrated the importance of cohort aggregation, minimum cell-size thresholds, complementary suppression, precision reduction, top and bottom coding, and re-identification risk assessment to support safe dissemination of analytical outputs.
  • Cohorts become the primary unit of statistical analysis. Transitioning from provider-level analytics to cohort-level statistical reporting preserved the ability to identify geographic patterns, benchmark peer populations, and monitor emerging trends while maintaining compliance with disclosure protection requirements.
  • Sharing data across agencies is complex. Federal data sources are used for different purposes and adhere to different disclosure rules, which may conflict and not be easily understood. Clarity on how to navigate data access and sharing across statistical and non-statistical Federal agencies can enhance project success.

 

Click here to learn more about this project

Disclaimer: America’s DataHub Consortium (ADC), a public-private partnership, implements research opportunities that support the strategic objectives of the National Center for Science and Engineering Statistics (NCSES) within the U.S. National Science Foundation (NSF). These results document research funded through ADC and is being shared to inform interested parties of ongoing activities and to encourage further discussion. Any opinions, findings, conclusions, or recommendations expressed above do not necessarily reflect the views of NCSES or NSF. Please send questions to [email protected].