Strabismus can be roughly defined as misalignment between the binocular axes, or 2 eyes appearing to look in different directions [1]. In a random photo, such gaze directions may rarely be accurately estimated; thus, a photo would not be reliable for determining whether the subject is at risk for strabismus. However, a simple point light source and the corneal luminous reflection it causes can make the problem noticeable.

Although the term “corneal luminous reflection” is technically inaccurate and the optical model is complex [21,22], the from-a-distance image of the corneal luminous reflection of a point light source can be approximated by a bright spot reflected off the surface of the cornea. Simultaneously, the gaze direction of that eye relative to the light source can be roughly reflected by the relative position of the center of the limbus [15], which is attached to the cornea, as illustrated in Figure 1. Combining both models, a simplified automated Hirschberg test can be achieved by taking a photo of the subject’s face with a point flashlight at a sufficient distance and comparing the relative positions of the center of each iris and the bright spot on it [22], as illustrated in Figure 2.

Smartphone photorefractory shares considerable similarity, in principle, with the automated Hirschberg test, but with a twist. They both work by analyzing the reflection pattern of a light source illuminating the eye. However, in smartphone photorefractory, the reflection occurs on the fundus instead of the cornea, and the pattern is primarily linked to the refractive error instead of the ocular axis, as illustrated in Figure 2. With no refractive error, the image of the light source on the fundus would be on its optically conjugate plane, and thus, it would be reflected back following exactly the same path and is almost invisible to the camera. Defocused by refractive error, the illumination would spread over the fundus and be reflected across directions and thus be partially captured by the camera. This is generally known as the red reflex [23].

Additionally, the size of the pupil affects the 2 phenomena. Although it does not interact with corneal luminous reflection, a dilated pupil admits more light into the vitreous chamber [24] and produces a wider viewing angle of the fundus [25]. As a result, when the pupil contracts to a minimal size, fundus reflection is subdued, for a high-quality automated Hirschberg test. When the pupil dilates widely, fundus reflection is prominent and can be better isolated from interfering with corneal luminous reflection. This is a critical consideration in designing the workflow that combines both tests.

The automated Hirschberg test can be numerically processed as follows. After extracting the positions of the limbus and the corneal luminous reflection spot from the captured image, the horizontal and vertical offsets of the center of the corneal luminous reflection spot relative to the center of the corresponding limbus are measured as Δx_L, Δy_L for the left eye and Δx_R, Δy_R for the right eye, respectively. These offsets are measured in pixels and signs following the computer vision convention, as illustrated in Figure 3 (top image). Assuming the frontal plane pixel scale is α mm per pixel, the eyeball diameter is d mm, and the gap between the camera and the flash is ignorable, the axial deviations of each eye from the camera/flash-ocular axis is as follows:

Where tan^–1() is the inverse tangent function. In this solution, d is unmeasurable and is thus set to an average of 20 mm [26]. In summary, the horizontal and vertical binocular axis deviations, measured in angles, are as follows:

Intuitively, significantly positive and negative θ_horizontal indicate exotropia, as illustrated in Figure 3 (top image) and esotropia (middle image), respectively. A significantly nonzero θ_horizontal indicates vertical strabismus, as illustrated in Figure 3 (top and bottom images).

The red reflex phenomenon, based on which photorefractory is implemented, is hard to robustly quantify with a point light source and uncalibrated parameters [27]. From the examiner’s perspective, assuming the light source is positioned to the left of the camera, a red crescent rising from left side of the pupil is associated with higher degrees of myopia and one from the right side indicates higher degrees of hyperopia, as illustrated in Figure 4. The presence of the crescent can be reasonably reliably detected as an objective and qualifiable signal for screening. The crescent width-severity relationship, however, requires smartphone model–specific calibration and may not hold across examinees; thus, it should be used only as a reference. Additionally, due to the eccentricity of the light source dot, no red reflex can be caused by either refractive error below threshold or a contracted pupil [25,27] (which is hard to detect without close examination). Further, both severe myopia and hyperopia lead to a fully lit-up pupil and are, thus, indistinguishable.

Although the smartphone-based automated Hirschberg test and photorefraction solution share the same corneal luminous reflection and red reflex principles with specialized devices, it lacks important features, including nonvisible and structured light sources, distance sensors, and a calibrated imaging system. Despite the exam accuracy being inevitably compromised, the applicability of the solution to screening and measurement automation is still obtainable using software. Sequentially in the acquired images, the head tilting and eye locations are detected by facial landmark recognition, the centers of the limbus are estimated by contour detection, the patterns of corneal luminous reflection and red reflex are estimated through shape fitting, and the frontal plane scale of pixels is estimated using an image analysis of the quick response code. It should be noted that processing would be done first on the automated Hirschberg test images and then on the photorefractory images for the former’s higher signal-to-noise ratio, which will be explained in the Discussion section.

As an alternative solution to the existing high-quality, full-service program with an automatic refractor exam and procedures by qualified optometrists, the proposed setup has been adapted to resource-limited rural schools with multiple modifications.

First, the fundamental technology may cover qualitative instead of quantitative analysis on the risk of strabismus, myopia, and anisometropia. Even though earlier research verified that such technology could produce reliable quantitative results, it is limited strictly to specific smartphone models, which are rarely available to the average user. Since the goal is to screen instead of diagnose, ease of use and accessibility dominate over a level of performance higher than necessary.

Second, the workflow should serve children as young as 4 years old. Although the application of smartphone photoscreening has been proven feasible to young children [18], their inability to understand instructions and reluctance to cooperate reduces the chances of successful measurement and thus prolongs measurement time. Therefore, pediatricians are consulted to optimize the workflow for young children’s psychological characteristics.

Third, the procedure should be executable by school teachers under loose supervision of a general practitioner (GP) with only remote training, yet the exam time per child should be no more than 1 minute. Although expertise may not be required to carry out the procedures, legal concerns and local customs require a GP, who services several villages that may be hours of travel apart, to perform the job. Additionally, on-site consultation provided by the GP must be part of the procedure.

Last, data analysis should be mostly automatic and nearly instantaneous. In China, optometric teleconsultation is under-staffed, and few optometrists are qualified to interpret photoscreening images. Giving on-site face-to-face feedback on the screening results not only builds trust and reduces the burden of follow-up phone calls to farmer parents with only a few hours of free time in the evening, it also creates an opportunity for offering other services, such as distributing premade eyeglasses.

Following the guidelines and the smartphone-based automated Hirschberg test and photorefraction principles, the system is outlined in Figure 5. The core image-processing algorithms are discussed in the previous subsection. The streamlined workflow is explained in the next subsection.

Figure 6 shows a typical screening room setup. Two children sit back-to-back with a teacher standing next to them (not illustrated). Dark curtains are hung between the children and behind the smartphones, the former to minimize interference between the 2 parallel tests and the latter to act as projector screens, described next.

Figure 7 illustrates the exam timeline from an illumination and image acquisition perspective, specifically the coordination of the camera with flash on the smartphone and a video projected onto the curtain in front of the children. When called, the children walk into the room and get seated, then start watching the video projected onto the curtain in front throughout the whole exam, with varying brightness. After several seconds of normal intensity, the video is dimmed and smoothly adapts the children’s eyes to the dark surroundings, thus dilating their pupils for more accurate measurement. After the video reaches its lowest brightness for 5 seconds to induce significant pupil dilation [31], the imaging procedure starts. With the smartphone camera, a short video is recorded during which a flash with maximum intensity is triggered. Followed by 5 seconds of the flash gradually brightening up from minimal in torch mode to contract the pupils in a controlled fashion, a still image is captured with another burst of a maximally intense flash, and the raw data acquisition for 1 child is completed. Since all equipment remains in place and the child sits still through the combined automated Hirschberg test and photorefraction with no intervention required, this is regarded as a one-step measurement. The first frame with maximal exposure of the video and the still image are used for red reflex and corneal luminous reflection pattern analysis, respectively, and the reasoning for this is explained in the Discussion section.

Figure 8 illustrates the workflow for serving 3 children in a streamlined fashion. On being called, a child enters the dark room and gets seated while a teacher enters the child’s personal information on a smartphone. After the adaptation period, the child is given the smartphone and instructed to hold it directly in front of their forehead while the image acquisition process begins on the capturing smartphone, as described above. On receiving the automatically analyzed result, a GP or nurse interprets the results to the teacher.

To evaluate the solution in real-world environments, experiments were conducted alongside physical exam programs in Luoyang, Henan, between September 1 and October 15. A total of 100 students aged evenly between 8 and 10 years from 2 schools were selected, 20 of whom were suspected by their teacher to have strabismus and the rest were randomly chosen among their classmates. The experiments were conducted in the school infirmaries under direct supervision of either the optometrists from the screening team or the school nurses. The smartphones used for capturing and analyzing data included 1 iPhone 5s (Apple Inc), 1 Honor 8 (Huawei Group Holdings Ltd), and 1 Mi 6 (XiaoMi Corp), all of which were low-end or outdated models at the time.

All 100 screened students successfully completed the data acquisition procedure within 2 rounds, and 87 of them passed with a single attempt. The failures were attributed to random eye blinking. Regardless of individual procedure success or failure, the streamlined workflow in Figure 8 was kept at an even pace, and each time slot took 19 seconds, on average. This translates into almost 200 students screened in 1 hour with just 1 supervising teacher and a consulting optometrist or nurse. With the projected cartoon playing, all students showed a high degree of compliance, and few complained about the exam duration or flashes. Outside the scheduled experiments, some 6-year-old students volunteered to participate, and none exhibited confusion or loss of concentration during the procedure.

After the smartphone-based exam, the screened students went through gold-standard optometry exams, and these results were recorded as the ground truth. The gold standard positive thresholds for strabismus, myopia, and anisometropia were chosen as at least 15Δ, at least –0.5D, and at least with a difference of 2D, respectively. The screening positives were defined as a crescent at least 1 mm wide, relative visual angle difference of 10°, and crescent width difference of at least 1 mm, respectively. Using the raw counts of true positives (TP), false positives (FP), true negatives (TN), and false negatives (FN), the statistical metrics accuracy, sensitivity, and specificity were defined as follows:

The raw counts of the experiment are summarized in Figure 9. Since myopia is diagnosed on a per-eye basis, its total count is double the number of examinees. It should also be noted that all 4 false negatives in the strabismus tests were attributed to the weakness of the Hirschberg tests in detecting hidden strabismus. The statistical metrics are tabulated in Table 1.

Although the solution produced satisfying accuracy with high efficiency in screening children in resource-limited areas, major improvements should be investigated in follow-up studies. First, the smartphone flash mechanism is unnecessarily counterproductive. The light-emitting diodes for flash used in modern smartphones are specifically designed to work in 2 modes: one consuming very high power in a short burst or limited power continuously. With a fixed exposure time to control motion blur, using the former mode reduces the camera’s sensitivity to light or analog gain and thus improves the signal-to-noise ratio in acquiring both automated Hirschberg test and red reflex images. However, modern smartphones use preflash to induce pupil contraction to suppress the red reflex phenomenon, more generally known as the red-eye effect. To circumvent this, the implementation app replaced the still-image application programming interface with the first intensely illuminated frame extracted from the video. This frame, though, is usually not illuminated with the highest intensity flash, and thus it suffers a reduced signal-to-noise ratio. If a future version of the smartphone camera control provides more flexibility in such control, image quality may be improved.

Second, the implementation software uses a nongeneralized limbus detection algorithm. The proposed method worked robustly on the Chinese students partly because they have a black limbus that is in high contrast to a white sclera. However, on some of the older students who wore colored contact lenses, the low signal-to-noise ratio of the images occasionally caused detection error.

In addition to the factors known to limit the performance of the proposed solution, further exploration is planned to address other areas for improvement. First, on the coordination between the smartphone and the projector: In the current version, these run independently and rely on an operator to synchronize them. A future version may include automatic synchronization on the smartphone side by detecting a predefined pattern of the projector, such as information encoded in color variation.

A second issue of interest is the video played during the exam. Some clips seemed to contain too much activity for some of the younger students to be able to maintain a fixed gaze on a consistent part of the screen, introducing unnecessary change in the visual axes. Some background music seemed better at attracting the students’ attention quickly. Detailed comparison and analysis, however, were beyond the scope of this preliminary study and will thus be left for follow-up studies.

The last area for future work is on carrying out the experiment on a larger scale and with different setups. The experiment reported in this paper was limited to 2 urban schools, and the equipment and personnel execution capability may be different from those in rural schools.

In this paper, we propose an effective and efficient smartphone-based children’s vision screening solution for areas with limited resources, implemented and evaluated. In a dark room, the examinee watches cartoons projected onto a dark curtain screen with program-controlled intensity, while a smartphone in front of the curtain runs the measurement program. The smartphone’s flash and camera are used as an eccentric point light source and image sensor, respectively, following the automated Hirschberg test and photorefractory principles. In the acquired images, the frontal plane resolution scale is automatically calibrated from a personal information–encoded quick response code displayed on another smartphone held in front of the examinee’s forehead. The examinee’s head gesture and eye location are estimated using a deep learning model. The limbus areas, interpupillary distance, corneal luminous reflection spots, and red reflex areas are detected by their corresponding algorithms to derive risk levels of strabismus, myopia, and anisometropia.

By splitting the workflow into 3 parts of inputting information, acquiring images, and automatically analyzing, interpreting, and consulting, the screening procedure is streamlined such that 1 GP supervising 1 teacher can screen, get an automatic diagnosis, and give on-the-spot consulting to 200 children per hour. The examiners need little training, and the equipment, including the model-neutral mainstream smartphones, is available to the average school. In the evaluation of 100 students between 8 and 10 years old, the high throughput and screening accuracy of the proposed solution were verified and recognized by participating optometrists.