Artificial Intelligence to Differentiate Pediatric Pseudopapilledema and True Papilledema on Fundus Photographs

Methods

We performed a retrospective multicenter review of fundus photographs of children with papilledema and pseudopapilledema submitted by 4 pediatric neuro-ophthalmologists (M.Y.C., S.L.P., G.H., and S.B.). This study was approved by the institutional review board at each separate institution and adhered to the tenets of the Declaration of Helsinki and the United States Health Insurance Portability and Accountability Act of 1996. Informed consent was waived by the institutional review board for this retrospective study. Children with papilledema and pseudopapilledema were identified either through an electronic medical records search using International Classification of Diseases (ICD) codes (papilledema: ICD-9377.0, 348.0, ICD-10 H47.11, G93.2; pseudopapilledema: ICD-9377.24, 377.21, ICD-10 H47.33, H47.32) or by searching an institutional database of patients with these diagnoses. Charts were reviewed by the submitting neuro-ophthalmologist to determine if patients met inclusion criteria for the study: age < 18 years at the time of fundus photograph acquisition, in addition to the following diagnostic criteria: children with papilledema were required to have elevated optic nerves on fundoscopy and/or OCT as well as neuroimaging evidence of intracranial pathology causing increased intracranial pressure (such as a brain tumor), or a lumbar puncture with opening pressure > 28 cm H₂O. Children with pseudopapilledema were required to have either normal neuroimaging (with normal brain parenchyma, no signs of hydrocephalus, mass, or structural lesion, and no meningeal enhancement) and lumbar puncture with normal opening pressure (< 28 cm H₂O), or longitudinal follow-up for ≥ 6 months demonstrating no change in optic nerve appearance. Longitudinal follow-up was required even in cases where optic disc drusen were demonstrated on ophthalmic imaging. We excluded cases with papilledema superimposed on pseudopapilledema. We also excluded patients with resolved papilledema/gliosis and optic atrophy, as well as fundus photographs that were taken using wide-field fundus cameras and photographs deemed by the submitting pediatric neuro-ophthalmologist to be too poor in quality for accurate classification.

Results

We included 851 fundus photographs of 235 children (105 with papilledema and 130 with pseudopapilledema). The demographics of included patients are shown in Table 1. The causes of papilledema and pseudopapilledema are provided in Table S1. The mean age at diagnosis was older in patients with papilledema (11.7 vs. 10.2 years, P = 0.0028). Females predominated in both groups, with no difference in sex distribution between groups. There was no difference in race or ethnicity between groups. Overall, the majority of patients were White, but Hispanic patients represented a significant minority (31%). There was a significant difference in distribution of patients by site, with University of California, Los Angeles and Stanford contributing relatively fewer patients with papilledema than the other sites. The most common cause of papilledema was pseudotumor cerebri syndrome (78%). The most common cause of pseudopapilledema was optic disc drusen (61%). Among patients with papilledema, 83 (79%) had lumbar puncture results available (in others, lumbar puncture was contraindicated due to a space-occupying lesion in the brain); the average opening pressure was 40 ± 10 cm H₂O.

Of the 851 fundus photographs, 380 were from children with papilledema and 471 were from children with pseudopapilledema. The distribution of fundus cameras used to acquire these photographs by site is shown in Table S2. Different fundus cameras were used at each site to acquire mydriatic or nonmydriatic, nonwidefield fundus photographs.

The receiver operating characteristic curve and accuracy, sensitivity, and specificity of the cross-validation model with various papilledema probability cut-off values are shown in Figure 1. The AUC on the cross-validation set was 0.77 (95% confidence interval 0.74–0.80). With a papilledema probability cut-off value of 0.45 (ie, probabilities ≥ 0.45 were classified as papilledema, and probabilities < 0.45 were classified as pseudopapilledema), the accuracy, sensitivity, and specificity were 70.0%, 73.4%, and 67.3%, respectively (Table 2). The AI model achieved significantly higher accuracy and sensitivity than expert 2 (70.0% vs. 61.0% and 73.4% vs. 62.9%, P < 0.0001 and P = 0.0003, respectively) on the cross-validation set. Positive and negative predictive values were also significantly higher than expert 2 (64.4% vs. 55.6% and 75.8% vs. 66.5%, respectively; P < 0.0001 for both comparisons). Specificity was not significantly different from experts 1 and 2 after correcting for multiple comparisons (67.3% vs. 60.3% and 59.4%, P = 0.02 and P = 0.008, respectively).

The performance of the AI model on the external test set is also provided in Table 2. The model achieved higher sensitivity, negative predictive value, and overall accuracy in the external test set compared with the cross-validation set. The AUC for the external test set was 0.81 (95% confidence interval 0.74–0.87). The accuracy, sensitivity, and specificity were 73.9%, 90.4%, and 56.4%, respectively. On external testing, the AI model was significantly more sensitive at detecting papilledema than both human experts (90.4% vs. 68.7%, P = 0.0002). Specificity did not significantly differ between the AI model and expert 1 or 2 on the external test set (56.4% vs. 61.5% and 65.4%, P = 0.49 and P = 0.27, respectively). Negative predictive value was higher in the AI model compared with both experts on the external test set (84.6% vs. 64.9% [expert 1] and 66.2% [expert 2]), although this difference did not reach statistical significance after correcting for multiple comparisons.

In the external test set, there were only 8 false negatives (i.e., fundus photographs of papilledema incorrectly classified as pseudopapilledema). Of these, 6 had correct classification of the fellow eye as papilledema. The single patient with papilledema whose fundus photographs of both eyes were misclassified as pseudopapilledema was a 16-year-old girl with pseudotumor cerebri syndrome and grade 1, nearly resolved papilledema (Figure 2). Thus, the model did not misclassify any patients with sight-threatening papilledema or papilledema from a life-threatening etiology.

To determine the effect of papilledema grade, performance metrics of the AI model and expert graders were repeated in subgroup analyses of photos demonstrating mild and moderate-to-severe papilledema. The distribution of papilledema grades is shown in Table S3. Mild papilledema (grade 1–2) was demonstrated in 204 (54%) photographs, and moderate-to-severe papilledema (grade 3–5) was seen in 176 (46%) photographs. Table 3 shows the accuracy, sensitivity, specificity, and positive and negative predictive values of the AI model and human experts in classifying fundus photographs of mild and moderate-to-severe papilledema. The sensitivity of the AI model in detecting mild papilledema was 64.2% in the cross-validation set, higher than both human experts (52.0% and 38.7%, P = 0.006 and P < 0.0001, respectively). The AI model’s sensitivity on the external test set was 87.8%, significantly higher than both human experts (53.1% and 49.0%, P = 0.0001 and P < 0.0001, respectively). Both positive and negative predictive value were higher in the AI model than both human experts in the cross-validation set, and the AI model also had significantly higher negative predictive value than both experts on the external test set for mild papilledema (88% vs. 67.6% and 67.1%, respectively; P = 0.0002 for both comparisons). The AI model and human experts had similar specificity when classifying mild papilledema. Additionally, the AI model and human experts performed similarly in accuracy and sensitivity when classifying moderate-to-severe papilledema. On the moderate-to-severe cases in the external test set, specificity was lower in the AI model compared with human experts (56.4% vs. 61.5% and 65.4%), but this difference did not reach statistical significance (P = 0.49 and P = 0.27).

A final subanalysis was performed including only the fundus photographs from each patient’s first ophthalmology visit (Table S4). The AI model (on both cross-validation and external test sets) had higher sensitivity, negative predictive value, and accuracy on this subgroup analysis compared with human experts, although differences did not reach statistical significance with the Bonferroni correction.

Finally, class activation maps highlighting the regions of each fundus photograph used by the AI model for classification were generated. Figure 3 displays an example of a fundus photograph of a child with pseudotumor cerebri that was incorrectly classified as pseudopapilledema by the AI model but correctly identified as papilledema by both human experts. Peripapillary hemorrhages, a sign of true papilledema, were cropped by the automated algorithm for optic nerve detection (Fig 3B). The class activation map (Fig 3C) indicates that the model focused primarily on the temporal optic nerve to make the incorrect classification decision, rather than the optic nerve borders or peripapillary region.

Discussion

In this study, we developed a deep learning AI model to differentiate pediatric papilledema from pseudopapilledema using transfer learning and a tribranch convolutional neural network. On external testing, the overall model achieved an AUC of 0.81. With the threshold optimized to maximize sensitivity while maintaining specificity similar to human experts, the accuracy, sensitivity, and specificity of the model on external testing were 73.9%, 90.4%, and 56.4%, respectively. The model was significantly more sensitive at detecting papilledema than human experts, and the difference between AI and human experts was especially marked in cases of mild (grade 1–2) papilledema. We chose to optimize sensitivity given that the consequences of misdiagnosing papilledema (potentially missing a life-threatening diagnosis such as a brain tumor) are more serious than mistaking pseudopapilledema for papilledema (resulting in unnecessary diagnostic work-up including neuroimaging and/or lumbar puncture). Importantly, in the external test set of 161 photographs, there were only 8 fundus photographs of papilledema that were misclassified. When these misclassifications were evaluated on a patient basis, there was only 1 patient with papilledema whose fundus photographs from both eyes were misclassified. This patient had nearly resolved papilledema from pseudotumor cerebri. Thus, on external testing, our model did not misclassify any patients with life- or vision-threatening papilledema.

The accuracy of our human graders (61.0%–67.1%) is similar to previous reports in the literature. In 2017, Chang et al² conducted a prospective study in which 3 masked expert neuro-ophthalmologists graded fundus photographs (and other imaging modalities) as representing pediatric pseudopapilledema or papilledema. The accuracy of classifying fundus photographs ranged from 63% to 71% for the 3 individual graders. The authors concluded that no imaging modality interpreted in isolation, including fundus photographs, achieved sufficient accuracy for differentiating pediatric papilledema from pseudopapilledema.

In contrast, the BONSAI group developed an AI model that detected papilledema on fundus photographs with an AUC of 0.96 and accuracy of 87.5% in the external testing data set.⁹ Major differences between the current study and the BONSAI study include (1) the average age of patients in the BONSAI study was 48.6 years, whereas our study included only pediatric patients; and (2) the BONSAI model was trained to differentiate photographs of papilledema from normal optic nerves and optic nerves with any other abnormality (including optic atrophy and congenital optic disc anomalies). Pediatric pseudopapilledema, as indicated by its name, is more similar in appearance to papilledema than normal optic nerves and other optic nerve pathologies. Therefore, the classification task for our model was more difficult than the BONSAI study and lower accuracies are expected. The difference in task difficulty is reflected in the difference in performance of human experts in our study compared with the BONSAI study (accuracy 61.0%–67.1% in our study vs. 89.0%–89.6% in the BONSAI study).¹⁴ Other smaller studies reported similarly high accuracy when using AI to detect papilledema on fundus photographs,15, 16, 17 but these studies also used variable comparison groups (including normal optic nerves) and did not report patient ages.

The most challenging clinical scenario in evaluating children with apparently swollen optic nerves is differentiating mild papilledema from pseudopapilledema. The AI model proved to be particularly helpful in this situation, as the sensitivity for detecting mild papilledema was 87.8% on external testing, clinically and statistically significantly superior to human experts (53.1% and 49.0%, P ≤ 0.0001). Both the AI model and human experts achieved high sensitivity in detecting moderate-to-severe papilledema (94.1 vs. 91.2% and 97.1%, P = 0.65 and P = 0.56, respectively).

Despite the AI model’s superiority to human experts, the accuracy and sensitivity are still insufficient for its use as the sole factor in deciding whether to pursue work-up for papilledema. Because we used a deep neural network, the image characteristics used by the model for classification are largely unknown. However, 1 possible factor contributing to reduced model accuracy may be postulated from the class activation map in Figure 3A. This fundus photograph of papilledema was incorrectly classified as pseudopapilledema and the salient features for human interpretation (peripapillary hemorrhage and obscuration of blood vessels) were missed by the model due to excessively tight cropping. These findings suggest that our model may benefit from adjusting the cropping area after the optic nerve is detected by the unsupervised algorithm.

Given that our model was particularly sensitive in detecting mild papilledema, we propose that AI may be useful in the future to triage children with apparently mildly swollen optic nerves prior to evaluation by pediatric neuro-ophthalmology. Those with a high probability of true papilledema may be referred for immediate assessment in the emergency department, whereas those with a lower probability may be considered for urgent outpatient evaluation. Moderate-to-severe papilledema is likely to be accurately detected by both humans and AI, although not all providers may reach the level of expertise of our graders.

Our study highlights the need for additional research on methods to improve our ability to differentiate pediatric papilledema from pseudopapilledema. The next step for the Pediatric Optic Nerve Investigator Group is to initiate a multicenter, multimodality prospective study to evaluate whether a combination of imaging techniques (ultrasonography, autofluorescence, OCT, fundus photography, and fluorescence angiography) and AI can accurately distinguish between the 2 diagnoses.

The strengths of this study include a relatively diverse data set with regards to ethnicity (31% Hispanic), acquisition of fundus photographs using multiple types of mydriatic and nonmydriatic fundus cameras and multiple different operators, and strict inclusion criteria minimizing the possibility of mislabeling the "gold standard" clinical diagnosis. All patients with papilledema had abnormal neuroimaging or elevated lumbar puncture opening pressure and all patients with pseudopapilledema either had normal neuroimaging and lumbar puncture opening pressure or no change in optic nerve appearance over at least 6 months. While the latter criterion may not absolutely exclude true papilledema, the average follow-up time in the pseudopapilledema patients was 28 months. After 2 years, some degree of atrophy or gliosis would be expected if these patients had true papilledema.

The limitations of this study include the retrospective nature and relatively small sample size. An automated method of detecting the optic nerve region was developed and used for this study, but this algorithm failed in a small subset (< 15%) of photographs, which required manual cropping. Refinement of the optic nerve detection algorithm is needed prior to AI model deployment for clinical purposes. Finally, the fundus photographs were submitted by pediatric neuro-ophthalmologists at tertiary care academic centers and may not be representative of the distribution and underlying causes of pediatric papilledema and pseudopapilledema in the general population. Specifically, the number of patients with papilledema was nearly equal to those with pseudopapilledema in our cohort, whereas pseudopapilledema is far more common (up to 10 times more frequent) in the general pediatric population.¹⁸ Although the positive predictive value of the AI model in our external test set was 69%, the positive predictive value would be much lower (18.5%) if the sensitivity and specificity of our model was applied to the general population with an estimated 10% prevalence of papilledema among children referred for elevated optic nerves. The skewed distribution is likely related to the recruitment sites, which were all highly subspecialized pediatric neuro-ophthalmic clinics at tertiary care referral centers, as well as a possible bias toward taking fundus photographs in patients with papilledema in order to monitor for changes. Future iterations of this AI model will benefit from inclusion of images from general pediatric ophthalmology clinics with a distribution of papilledema and pseudopapilledema photographs that is more representative of the general population.

In conclusion, we report that an AI model can detect papilledema with higher accuracy and sensitivity than human experts and may be particularly helpful in differentiating mild papilledema from pseudopapilledema. However, further studies with additional data, ideally including a more representative ratio of papilledema to pseudopapilledema images, are needed. Larger datasets will improve the accuracy of the AI model beyond this proof-of-concept, pilot study. A future multicenter, multimodality prospective study incorporating AI may further advance our ability to differentiate between pediatric papilledema and pseudopapilledema, reducing the likelihood of missing space-occupying lesions in the brain or subjecting children to unnecessary and invasive procedures.

Manuscript no. XOPS-D-23-00120.

Supplementary Data