top of page
Design Process
1/Discovery
Problem
Design question:
When Microsoft first came to us, we were given a big and super difficult task:
How can we cheaply detect diabetic retinopathy in developing nations like India?
Problem space
What's DR(Diabetic retinopathy)?
Normal retina
Retina with DR
This is when high blood sugar levels cause damage to blood vessels in the retina. These blood vessels can swell and leak. Or they can close, stopping blood from passing through. Sometimes abnormal new blood vessels grow on the retina.
As the condition progresses, diabetic retinopathy symptoms may include:Spots or dark strings floating in your vision (floaters), Blurred vision, Fluctuating vision, Impaired color vision, Dark or empty areas in your vision, Vision loss.
source:
https://www.mayoclinic.org/diseases-conditions/diabetic-retinopathy/symptoms-causes/syc-20371611
https://www.aao.org/eye-health/diseases/what-is-diabetic-retinopathy
DR in developing countries
In developing countries the high cost of devices to diagnose diabetic retinopathy is prohibitive, and patients are unlikely to travel those distances to seek out medical help. Diabetes is projected to affect over 366 million people in 2030 according to the WHO [1]. Diabetes can lead to eye diseases such as diabetic retinopathy, cataracts, and glaucoma. Furthermore,
diabetes is the 5th leading cause of all blindness in the world [2]. The global prevalence of diabetic retinopathy is 34.6% affecting over 95 million people [3, 7]. When diagnosed and treated early, the risk of going blind decreases by 95% [8].
Redefine design question
After having a brief understanding of the problem, we had a discussion with Microsoft team to redefine the design question to give more eye care access to people in developing nations.
But.........
look at our team, no one has an optical physics background.
So in order to approach the problem properly and come up with design features from scratch, we started a first-round research including secondary research, technology research both for software and hardware, competitive analysis, field research, expert interview, surveys, end user interviews.
2/Define
User need
In this stage, we made decision mostly based on expert’s advice and better user experience.
Competitive analysis
After understanding the problem a little more, we had a meeting with Microsoft. In this meeting, we together brainstormed several possible solutions and options, for example, we can use a phone camera directly or a standalone camera system to generate flashing light. In order to make proper design decision, we investigated both the advantages and disadvantages of all possible solutions.
Technology research
Hardware
-Optics & lenses
-Headset design & facial structures
-Mobile phone & RPi cameras
-IR & RGB imaging
Software
-Improving image quality with GANs
-Computer vision techniques
-Multispectral imaging
-Videography
Possible solutions
1/ Phone camera vs. No phone camera
2/ One eye vs. Two eye
3/ Infrared light vs. White light
4/ Dilation vs. No dilation
5/ Take video vs. Take multiple pictures.
Field research
No dilation
-can't see the fundus
-Very bulky but stable
-Intense white light
-Need professional photographer
With dilation
-can see the fundus
-Medium size
-Laser light
-Need professional photographer
With dilation
-can see the fundus
-small size but very unstable
-Intense white light
-Less professional
Expert interview
Aaron Y. Lee M.D.
Ophthalmology
(eye disorders)
-Attach lens to the phone, it can be very small, but it can only see retina with small field of view like 2-3 degree. Phone camera has an infrared filter filtering out the infrared light.
-Expert level is 240 degrees which is the entire retina. At 130 degree macula centered. 1 degree is 2-3 mm.
-We need to reach 60-70 degrees field of view to reach to 95% of the population screening.
-Fovea is the center of the macula which is an area of the retina. Diabetic retinopathy firsts affects the periphery.
-Most eye diseases we have almost balanced eye issues.
-Two key problems of existing landscape: persons hand and patient is moving. Patient does not know where to look. We can direct gaze with the opposite eye, take images of the other eye.
-We can use infrared light source to capture image instead of the white light.
-User experience is important!
-“Human’s eyes are very sensitive to the intense white light, so we have to dilate patients most of the time.”
Ramkumar Sabesan
-We can start with a linear structure to understand basic optical and retina knowledge.
-The biggest challenge of the project is how to reduce glare from the cornea and provide a bigger field of view with safe but cheap material and components.
-Image is 256x256 should be sufficient for the models. 15 degree field by 1536x1536 pixels is minimally sufficient for a doctor. A 5MG raspberry pi camera is sufficient (5 MP- 2560x1920 resolution)
-Polarizers and beamsplitters can help reduce glare.
Ophthalmology
Vision Research Scientist
Eye screening experience survey
Since Dr. Lee mentioned that user experience is very important, we want to know what's current eye screening process and its advantages and disadvantages.
We sent out 2 surveys, one for normal people, the other for diabetic patients. Although we did it through Mturk and some of the results is not convincing, we can still find out some common pain points in the survey.
Likes / good experience
Dislikes / bad experience
End user interview
Dr.Guo
42, he is an experienced ophthalmologist who volunteered to help diagnose eye diseases in Africa for one year.
-People have various eye diopters. Patients with myopia, hyperopia and astigmatism all have different diopters, so doctors need to change the focal length of the equipment.
-Most of the normal people’ fundus photos can be taken in the dark time, but some people can’t, including people with astigmatism and cataract. People with diabetics are more likely to have astigmatism and cataract and the radius of the pupil in the dark is smaller than ordinary people (diabetics will affect the muscle around the eye ), so they need to be dilated.
-It can be troublesome when some of the equipment are broken because there are not sufficient professional maintenance workers.
-The Internet there is good enough to send pictures and receive basic information and images.
-Most people are not aware that they have more than one eye diseases, so it’s be hard to diagnose for the first time. And even worse, sometimes they only came to the camp once and never showed up anymore even though they just received a cataract surgery.
-For giving instructions to untrained volunteers, it is very important to first determine what is the cause for the blurred images, is that because of the equipment, their hands or the patients.
Summary- Stakeholder analysis - Design requirement
Design feature v1
With these design requirements, we summarized some basic design features.
However, there were also many constrains, including limited processing power, cheap components, unavoidable intensive white light, limited field of view, not self-help product, less precise and limited adjustment range.
Although we have came up with these design requirements, we still felt blank about how to actually take a fundus image. We gradually found out that the first priority of this project is to have a working hardware or camera. We started with doing experiment using an open-source product "O-Docs".
First Failure
It said that O Docs can take clear fundus images, however.....
The problems are:
1. This device is not stable at all, both doctors' hands and the patient's eye are shaking and moving.
2. Tons of glare and cornea reflection because of its linear structure.
3. The image quality was high enough and it's too small.
Preliminary design review
With this prototype zero, we had a preliminary design review with Dr. Ramkumar and Dr.Lee. In that meeting we worked collaboratively out a new T-shape lens system to better reduce glare and improve image quality. In this new design, distance between different components are the most important.
Linear polarizer: only let vertical or horizontal light pass,
filter out one direction.
Beamsplitter:splits a beam of light in two.
Condensing lens: converging beam to illuminate an object
Second failure
In addition, in the following hardworking weeks, we also found that IR (Infrared) light is less painful to the eyes, but the lens system reducing glare generated from IR light is different from that from white light. What's worse, if we only use infrared light, it can't be recognized by Microsoft's machine learning model, which is not ideal.
So, we determined to use IR light for alignment(focus on the right spot) and take RGB images with a flashing white light.
Third failure
Although, we have already figured out the ideal T-shape lens systems, but in reality, we still get a lot reflection and we can't focus on the right distance with prototype 2.0. So we still needed to improve the design and the prototype.
Luckily, we have another mentor Joel Kollin from Microsoft to help us, we had a separate hardware meeting with him every week in Microsoft to work together on getting a high resolution fundus image.
1/Camera needs to be sensitive to the 810 nm(near-infrared) range.
2/1.7 mm lens is too broad, we need a smaller field view.
3/The current RPi camera can be tried with different M-12 lens to test, such as 6mm or 8mm.
4/80-20 optical bench can be helpful to test our optical system
5/IR intensity is too low. Originally 10 mW of intensity coming from 810 nm light but only .1 mW reaches the eye.
Joel Kollin
Principal Research Hardware Development Engineer at Microsoft
Design feature v2
But the good news is, the image is way better with the new design, so our had a new design features and touch point analysis to work on software simultaneously.
Storyboard
With the new design feature, I draw a storyboard of the main process.
1/Get training before taking image
2/Do alignment in dark with infrared light
3/Click on "Take image" to activate white flashing
4/Upload to cloud to do classification
5/Send back diagnose results
4/Design
Iterations
Prototype design
Iteration 2
Tested on table to see how clear the image we can get. At the same time, we did some math to find out the proper distance.
Iteration 3
Industrial design
UI / picture of the front of the eye
Likes
1/All the components can be detached to small modules.
2/To make it stable, I made a holder to let it stand alone.
3/I made thread on the camera, it can move back and forth to adjust the focus position of the camera.
4/We also have a rough real-time video frame on raspberry pi.
5/Look at the picture we got, even it is only the front of the eye, but the resolution, glare were way better than before.
Problems
1/ Instead of moving the whole camera, better solution is to change the distance between the lens and the filming sensor. But adjusting the inside lens from outside enclosure is not easy.
2/ We still can't see through the pupil to the fundus, in order to solve the problem, we decided to follow the advice from Joel, to take every part out and place it in an optical bench for more precise adjustment.
First evaluation
With the first working prototype, we scheduled the first evaluation.
Goal: it mainly focused on testing whether users can understand how to use the prototype and guide the patients to see different guide lights to take a good front eye image.
At the same time, we wanted to know what our UI features should we provide to help take images.
Method:
1/In order to let them know what a good image, we gave the participants 4 sample images with different guidelights to re-create and capture.
2/We recorded adjustments, errors, comments, and suggestions.
3/Recruited 8 participants to capture images with our prototype.
Results:
Average time: 5 min
(range of 1m40s-10m)
Average error: 32.5%
(range of 8-41%)
There is large deviation which indicates different understanding of the prototype between different people .
We compared the image quality based on the euclidean distance between features of the sample images and their images.So there is a gap between the quality of the images from our baseline sample images.
Evaluation result
1/Process of setting up and taking a picture was easy, alignment was hard.
2/Orientation of guide lights are confusing
3/Users want to know or confirm the quality of the image they have taken.
1/User’s should be taught what is type of picture is acceptable.
2/Guide lights help keep the patient’s eye stable
3/Moving the camera was intuitive
Insights:
Give real-time instructions and guidelines while taking image
User needs confirmation of whether the image quality is acceptable
X,Y,Z Rail slide for adjustment
Inner camera lens focus adjustment
Remove circular glare patterns
Image enhancement using SR-GAN, pix2pix GAN with ground truth from clinical fundus cameras
Iteration 4
Finally!!! Fundus Image!!! 🎉
After putting all the components in an adjustable optical bench, we changed the distances between components and each component's angle more precisely with the help from Joel. In this professional optical bench, we got our first fundus image although with dilation.🎉🎉🎉
We have figured out the right distance and components for the hardware so far, what left to do is to design the companion app and put all the components into an enclosure.
Key findings:
Q: What adjustments are necessary to see the fundus of different patients?
A: Besides from the camera focal length and the distance from the objective lens to the eye needs to be adjusted. The angle of the beamsplitter also need to be adjustable in a small range for glare reduction.
Q: What is the acceptable range of measurements to repeatedly gather a clear fundus image?
A: 38-44mm from the objective lens to the front of the eye. Camera needs to be focused at 110mm.
In order to provide better user experience as expected, we decided to conduct another round user evaluation.
Second evaluation
With the first working prototype, we scheduled the second evaluation.
Goal:
It mainly helped us find out whether the optical layout enable non-medical professional to capture a standard quality fundus image and test whether the images can be used for processing.
Method:
1/Based on the findings from the Evaluation 1, "User needs confirmation of whether the image quality is acceptable", we developed a simple imaging instructions software which integrated in form of standard image overlay, quality measure and confirmation.
Results:
Average time:
6 mins 10s vs 5 mins
Image moves too fast due to instability
Average error: 20% vs 32.5%
It is helpful with the confirmation and the overlaying sample images
1/More guidelines are necessary, for example participants found it helpful to be able to start from back and enter the pupil. A demo exercise will be helpful.
-UI
2/Confusion about optical disk and different parts of the eye they are seeing.
--Training
3/ At the same time, the field of view this prototype provide is bigger than before but still less than 60-70 degree.
Overall findings
Companion app
Software
Prototype
1/ Give volunteers some basic training on what's optical disk, what's fundus image, how to focus the camera on the pupil before taking actual images.
2/ Give real-time feedback and guidelines while taking image. Imaging instructions integrated in form of standard image overlay is a good option.
3/ Give confirmation after taking the image.
4/ Switch between infrared led and white light led automatically with script.
5/ In order to satisfy the requirement of 60-70 degree of view, we have to take 2 images, one optical disk centered, one fovea centered and stitch them together with ML algorithm to have a whole fundus image. So, we have to place two guide lights horizontally in a small distance inside the enclosure for patients to stare at and keeping their eyes still.
6/ X,Y,Z Rail slide for stable adjustment.
7/ Able to adjust the focal length of the lens and the angle of the beamsplitter from the outside of the enclosure.
5/Design
Solution
Up to now, we had our final solution, it is a very complicated hardware, software and UI system. I met many difficulties in designing UIs. In order to summarize design features and information architecture of the webapp, I first went through our solution with design features and touch point analysis
Storyboard
Touch point analysis
Information architecture
4/Design
Iterations
Wireframe
With the information architecture, I designed several wireframes.
UI design v1
With wireframes, I designed first version of the UI with Google material design guidance
Training
Image taking
Usability test 1
Goal:
It mainly tested with non-medical professionals and test their user experience of capturing images.
Method:
1/ Recorded mistakes made
2/ Asked their understanding and opinions on the image taking process.
3/ Give scores on the difficulty of using this app for untrained volunteers
4/ Evaluation plan: https://docs.google.com/document/d/1IuJtark633ueXjMc9ag_RNGvRBlW-lfuWD9mA7T6lIo/edit?usp=sharing
Results:
Average time:
Average time to complete the entire process (UI only) was under 6 minutes
Average error:
On average 0.4 mistakes per person
1/They all think the process is easy to understand and straightforward. The average difficulty score is 3.7 (1 is the easiest, 10 is the highest)
2/Nice information architecture and the interaction is simple.
3/ Giving quizzes after introducing concept as a method of training is very helpful. The quiz scores were high, all participants scored 100/100.
Problems:
1/The image of showing live video and the target image at the same time is not intuitive for users, but they can understand easily how to use it after explanation. In addition, they suggest that we can also show some simpler and easier instructions to help the users do alignment.
2/ They didn't really know which step they currently are, they felt confused when they saw the progress bar. Our system architecture is very complicated, I need to figure out another way to better tell users where are they now
3/ Two third of the participants want to review the images they took, and they want to know what’s going on in each “processing” page. However, Our goal is improve ease of use for the non-professional users and reduce processing for our system.
4/ They wanted more confirmation in image taking process.
5/ Average mistakes in all the training sections was 1.5 mistakes per participants. The most common mistakes they made were in the first page where they misunderstand the step to be clickable checkbox. -- UI mistakes
A/B Test 1
In our system, we have 5 big steps and 4 sub steps.
1/ Fill out patient information;
2/ Take first optical disk centered image1;
2.1 see sample images, 2.2 Do alignment with infrared light, 2.3 Take images with white light, 2.4 Confirmation
3/ Take second fovea centered image2;
2.1 see sample images, 2.2 Do alignment with infrared light, 2.3 Take images with white light, 2.4 Confirmation
4/ Stitching image1 + image2;
5/ Get diagnose result.
In order to figure out the best way to show the progress, I brainstormed 4 different ways and conducted an A/B test.
Big step: progree bar, sub steps: dots
Big step: step number, sub steps: dots
Big step: Text, sub steps: dots
Big step: step number, sub steps: Text
I asked 8 people's opinions on which one is the most clear one, they love text more but their preference varied. we finally found out the best one is the integrate one, which shows as follows:
And figured out the best way is to have
a step number showing big steps on the top and a small indicators showing substeps in the bottom. And at the same time have big text on the page to tell the step.
A/B Test 2
We also did an A/B testing on 3 ways to review the image because there is an inconsistent expectation between users and our goal. Our goal is to improve ease of use for the non-professional users and reduce processing for our system, so I didn't encourage users to review them which costs some processing power.
Show images directly on the left
Show images with a obvious button which is in the same importance level.
Show images with a less obvious gray text
In the A/B test, some participants thought the first one shows too much information. Compared to 2, 3 shows a sense of our goal which means I can understand that I am able to review the image if I want but I am not encourage to do so. In addition, they thought that after some practices, as volunteers, they don't really need to review the image every time.
So, in order to achieve our goals and hit the user pain points, we will allow users to review images through a less obvious “See Image” button, option 3.
Major UI design improvement
Based on the finding from first usability test and A/B testing, I made the following improvement accordingly.
01
"Showing live video and the target image at the same time is not intuitive for users, although they can understand easily how to use it after explanation.
They suggested that we can give simpler and easier instructions to help the users do alignment."
Instead of doing the sample images vs live video, we give them two different options to switch on and off:
1/Guide circles to only show the place of optical disk or fovea.
2/Transparent sample images to give a more detailed instructions
02
Confusion around the 5 big steps and each of their corresponding 4 substeps when taking actual images.
We did A/B testing on 4 ways to show steps and substeps, and which ones were easiest for the user to understand.
03
Two-thirds of users want to review the images they took.
Our goal is improve ease of use for the non-professional users and reduce processing for our system.
In order to achieve our goals and hit the user pain points, we will allow users to review images through a less obvious “See Image” button
04
Users require more indication of when the image is aligned so they know when to capture the image.
Only when the image is aligned will the “take image” option appear on the screen.
05
In the training section, The most common mistakes they made were in the first page where they misunderstand the step to be clickable checkbox.
5/Launch
Final presentation
Final User Interface
LOG in
Training
Search a patient
Final Prototype
-3.jpg)
bottom of page