Chirag Ghube
Explore
UX Designer & Web Developer
Chirag Ghube
Explore
Chirag Ghube
Explore
UX Designer & Web Developer
SoundSense: Smartwatch Sound Awareness App for D/deaf and Hard of Hearing Users
Tags
Accessibility, DHH, Smartwatch, Sound Awareness, Capstone Project
Client
Capstone Project
Completed
2023
Context
Enhance the quality of life for D/deaf and hard of hearing (DHH) individuals by providing them with an accessible and customizable way to detect and interpret sounds in their environment
Why do we need sound awareness systems?
DHH individuals often struggle to detect and interpret sounds in their environment, making it difficult for them to navigate their daily lives and engage with their surroundings. The lack of effective sound awareness technology was a major problem for those who were constantly struggling to detect and interpret sounds in their environment.
What are sound awareness systems?
Sound awareness systems are devices or software that help individuals detect and interpret sounds in their environment. The goal of these systems is to enhance the quality of life for DHH individuals by providing them with an accessible way to detect and interpret sounds.
This project was my graduate capstone project with RIT, NY.
Problem Statement
Provide design guidelines and an example for smartwatch based sound awareness system
Previous studies have explored sound awareness technology for this community, with key findings including the importance of personalization and user-centered design. However, there are still limitations in current technology that must be addressed to provide effective and usable sound awareness systems. Therefore, we believe that providing design guidelines and an example for smartwatch-based sound awareness systems will help further research and development in this area, ultimately improving the quality of life for DHH individuals.
Users
5 DHH participants using Assistive Listening Devices (ALD)
The five participants in this study were adults with ages ranging from 21 to 55 years old. Two participants identified as d/Deaf, two as hard of hearing, and one participant was deaf in their right ear. Two participants used hearing aids, and one participant used cochlear implants. All participants were from the Rochester area and had diverse backgrounds in terms of education and employment.
Semi-structured Zoom interviews with live captioning
The intention behind the interviews was to understand the unique needs and perspectives of the DHH community and to gather data that could inform the design of effective and usable sound awareness systems. Live captioning was used during the interviews to ensure effective communication, and the data collected was expected to provide insights into the design of smartwatch-based sound awareness systems that would enhance the quality of life for DHH individuals.
In-Vivo and Axial Coding
We used In-Vivo coding to identify key phrases and words used by participants, and Axial coding to categorize and organize data into themes. In-Vivo coding allowed analysis in participants' own words and Axial coding identified common themes. These techniques are commonly used in qualitative research and allowed for a thorough analysis of the data.
Solution
"I have difficulty figuring out where the sound is coming from."
Use motion to convey the direction of sound and intensity
P1 mentioned that they have difficulty locating the direction of sound, saying, "I have difficulty figuring out where the sound is coming from." P2 agreed, saying, "I feel like sometimes I can't tell if it's coming from the left or the right." Therefore, the system should provide information about the direction of the sound.
"I feel like if people had the ability to customize what sounds they want to appear on their smartwatch, that would be more beneficial."
Make sure you provide a way customize most elements of the UI
P2 said, "I feel like if people had the ability to customize what sounds they want to appear on their smartwatch, that would be more beneficial." This suggests the importance of providing users with the ability to customize the user interface. Therefore, the system should allow users to choose which sounds to prioritize and display.
"I want to be able to filter out certain sounds, like alarms."
Provide the ability to filter out sounds based on algorithm confidence
P2 mentioned that they want the ability to filter the kind of sound notifications they receive at any given time, saying, "I want to be able to filter out certain sounds, like alarms." P5 also mentioned that they would like a system that could notify custom sounds, like someone calling out their name or a particular alarm in their house. Therefore, the system should provide the ability to select specific sounds to detect.
"And also, if it were easier to dismiss the notification…”
Provide an easy way to dismiss and mute the notification
P5 described the current system as "time-consuming" and suggested that making it easier to dismiss notifications would be beneficial: "And also, if it were easier to dismiss the notification, you could say 'Okay, turn off the notification for half an hour or an hour' and do that in the watch or app itself. That would be nice as well." The participants prefer their notifications to be concise, i.e. glanceable, and easy to dismiss. Therefore, ensure that the notification fits on one screen and has a dismiss and a mute button.
Handling Unknown Sounds
Maintain a history of sounds that the user can access
As P5 stated, "maybe you could just let me know in the background. That would be nice. Background kind of report that I can access in real time, maybe. I could see, Okay, these are the sound that is heard, but it's not reporting to me actively." This suggests that users may value having access to information about unknown sounds without being actively notified. Therefore, the system should provide a list of identified sounds, with unknown sounds tagged as "Maybe", and display a sound report to provide more information.
"I have difficulty understanding what people are saying in a noisy environment”
Provide captions in noisy environments and group settings
P2 said, "I have difficulty understanding what people are saying in a noisy environment; that's really a much bigger issue." This feature would enable DHH individuals to better engage with conversations, enhancing their communication experience and reducing feelings of isolation. Therefore, the system should provide live captioning with a dismiss and mute button.
Connecting Assistive Listening Devices
Users should be able to control their ALDs and the programs on the ALDs should match the modes in the app
P1, P2, and P5 wanted a sound awareness system that could also control ALDs. The biggest pain point for ALDs is their batteries running out without warning. We can fix that by notifying users right before their batteries are about to run out. Most ALDs have different programs that provide various sound filters. The modes in the sound awareness system should synchronize with the programs to provide more control for the users. Therefore, the system should display the user's ALD information, including battery status, volume control, and power control. It should also display a notification when the battery in ALDs is running out.
Prototype
success measures
Usability Testing
We need to conduct usability testing with DHH users while they are using the app. We propose a Wizard-of-Oz study that will check the users response in 3 situations like a bus stop, cafe, and build lobby.
how did i change?
Through my graduate capstone project SoundSense, I gained valuable experience in designing technology for the Deaf and Hard of Hearing (DHH) community. Despite challenges in recruiting participants and communicating with DHH individuals during interviews, I was able to complete the project successfully and developed a comprehensive solution that addressed the lack of effective sound awareness technology for DHH individuals.
Through the project, I gained insights into the experiences and preferences of the DHH community regarding sound awareness technologies for smartwatches. I also discovered the sound programs of hearing aids, which I found fascinating. Overall, the project allowed me to grow in my understanding of the challenges and opportunities in designing technology for the DHH community. It inspired me to continue prioritizing inclusivity and accessibility in my work as a designer.
Chirag Ghube
Explore
UX Designer & Web Developer
Tags
Client
Completed
Deliverables
My Role
Context
Problem Statement
Users
Solution
Prototype
Recap
What could go wrong?
success measures
how did i change?
SoundSense: Smartwatch Sound Awareness App for D/deaf and Hard of Hearing Users
Tags
Accessibility, DHH, Smartwatch, Sound Awareness, Capstone Project
Client
Capstone Project
Completed
2023
Context
Enhance the quality of life for D/deaf and hard of hearing (DHH) individuals by providing them with an accessible and customizable way to detect and interpret sounds in their environment
Why do we need sound awareness systems?
DHH individuals often struggle to detect and interpret sounds in their environment, making it difficult for them to navigate their daily lives and engage with their surroundings. The lack of effective sound awareness technology was a major problem for those who were constantly struggling to detect and interpret sounds in their environment.
What are sound awareness systems?
Sound awareness systems are devices or software that help individuals detect and interpret sounds in their environment. The goal of these systems is to enhance the quality of life for DHH individuals by providing them with an accessible way to detect and interpret sounds.
This project was my graduate capstone project with RIT, NY.
Problem Statement
Provide design guidelines and an example for smartwatch based sound awareness system
Previous studies have explored sound awareness technology for this community, with key findings including the importance of personalization and user-centered design. However, there are still limitations in current technology that must be addressed to provide effective and usable sound awareness systems. Therefore, we believe that providing design guidelines and an example for smartwatch-based sound awareness systems will help further research and development in this area, ultimately improving the quality of life for DHH individuals.
Users
5 DHH participants using Assistive Listening Devices (ALD)
The five participants in this study were adults with ages ranging from 21 to 55 years old. Two participants identified as d/Deaf, two as hard of hearing, and one participant was deaf in their right ear. Two participants used hearing aids, and one participant used cochlear implants. All participants were from the Rochester area and had diverse backgrounds in terms of education and employment.
Semi-structured Zoom interviews with live captioning
The intention behind the interviews was to understand the unique needs and perspectives of the DHH community and to gather data that could inform the design of effective and usable sound awareness systems. Live captioning was used during the interviews to ensure effective communication, and the data collected was expected to provide insights into the design of smartwatch-based sound awareness systems that would enhance the quality of life for DHH individuals.
In-Vivo and Axial Coding
We used In-Vivo coding to identify key phrases and words used by participants, and Axial coding to categorize and organize data into themes. In-Vivo coding allowed analysis in participants' own words and Axial coding identified common themes. These techniques are commonly used in qualitative research and allowed for a thorough analysis of the data.
Solution
"I have difficulty figuring out where the sound is coming from."
Use motion to convey the direction of sound and intensity
P1 mentioned that they have difficulty locating the direction of sound, saying, "I have difficulty figuring out where the sound is coming from." P2 agreed, saying, "I feel like sometimes I can't tell if it's coming from the left or the right." Therefore, the system should provide information about the direction of the sound.
"I feel like if people had the ability to customize what sounds they want to appear on their smartwatch, that would be more beneficial."
Make sure you provide a way customize most elements of the UI
P2 said, "I feel like if people had the ability to customize what sounds they want to appear on their smartwatch, that would be more beneficial." This suggests the importance of providing users with the ability to customize the user interface. Therefore, the system should allow users to choose which sounds to prioritize and display.
"I want to be able to filter out certain sounds, like alarms."
Provide the ability to filter out sounds based on algorithm confidence
P2 mentioned that they want the ability to filter the kind of sound notifications they receive at any given time, saying, "I want to be able to filter out certain sounds, like alarms." P5 also mentioned that they would like a system that could notify custom sounds, like someone calling out their name or a particular alarm in their house. Therefore, the system should provide the ability to select specific sounds to detect.
"And also, if it were easier to dismiss the notification…”
Provide an easy way to dismiss and mute the notification
P5 described the current system as "time-consuming" and suggested that making it easier to dismiss notifications would be beneficial: "And also, if it were easier to dismiss the notification, you could say 'Okay, turn off the notification for half an hour or an hour' and do that in the watch or app itself. That would be nice as well." The participants prefer their notifications to be concise, i.e. glanceable, and easy to dismiss. Therefore, ensure that the notification fits on one screen and has a dismiss and a mute button.
Handling Unknown Sounds
Maintain a history of sounds that the user can access
As P5 stated, "maybe you could just let me know in the background. That would be nice. Background kind of report that I can access in real time, maybe. I could see, Okay, these are the sound that is heard, but it's not reporting to me actively." This suggests that users may value having access to information about unknown sounds without being actively notified. Therefore, the system should provide a list of identified sounds, with unknown sounds tagged as "Maybe", and display a sound report to provide more information.
"I have difficulty understanding what people are saying in a noisy environment”
Provide captions in noisy environments and group settings
P2 said, "I have difficulty understanding what people are saying in a noisy environment; that's really a much bigger issue." This feature would enable DHH individuals to better engage with conversations, enhancing their communication experience and reducing feelings of isolation. Therefore, the system should provide live captioning with a dismiss and mute button.
Connecting Assistive Listening Devices
Users should be able to control their ALDs and the programs on the ALDs should match the modes in the app
P1, P2, and P5 wanted a sound awareness system that could also control ALDs. The biggest pain point for ALDs is their batteries running out without warning. We can fix that by notifying users right before their batteries are about to run out. Most ALDs have different programs that provide various sound filters. The modes in the sound awareness system should synchronize with the programs to provide more control for the users. Therefore, the system should display the user's ALD information, including battery status, volume control, and power control. It should also display a notification when the battery in ALDs is running out.
Prototype
success measures
Usability Testing
We need to conduct usability testing with DHH users while they are using the app. We propose a Wizard-of-Oz study that will check the users response in 3 situations like a bus stop, cafe, and build lobby.
how did i change?
Through my graduate capstone project SoundSense, I gained valuable experience in designing technology for the Deaf and Hard of Hearing (DHH) community. Despite challenges in recruiting participants and communicating with DHH individuals during interviews, I was able to complete the project successfully and developed a comprehensive solution that addressed the lack of effective sound awareness technology for DHH individuals.
Through the project, I gained insights into the experiences and preferences of the DHH community regarding sound awareness technologies for smartwatches. I also discovered the sound programs of hearing aids, which I found fascinating. Overall, the project allowed me to grow in my understanding of the challenges and opportunities in designing technology for the DHH community. It inspired me to continue prioritizing inclusivity and accessibility in my work as a designer.
Chirag Ghube
Explore
UX Designer & Web Developer
SoundSense: Smartwatch Sound Awareness App for D/deaf and Hard of Hearing Users
Tags
Accessibility, DHH, Smartwatch, Sound Awareness, Capstone Project
Client
Capstone Project
Completed
2023
Context
Enhance the quality of life for D/deaf and hard of hearing (DHH) individuals by providing them with an accessible and customizable way to detect and interpret sounds in their environment
Why do we need sound awareness systems?
DHH individuals often struggle to detect and interpret sounds in their environment, making it difficult for them to navigate their daily lives and engage with their surroundings. The lack of effective sound awareness technology was a major problem for those who were constantly struggling to detect and interpret sounds in their environment.
What are sound awareness systems?
Sound awareness systems are devices or software that help individuals detect and interpret sounds in their environment. The goal of these systems is to enhance the quality of life for DHH individuals by providing them with an accessible way to detect and interpret sounds.
This project was my graduate capstone project with RIT, NY.
Problem Statement
Provide design guidelines and an example for smartwatch based sound awareness system
Previous studies have explored sound awareness technology for this community, with key findings including the importance of personalization and user-centered design. However, there are still limitations in current technology that must be addressed to provide effective and usable sound awareness systems. Therefore, we believe that providing design guidelines and an example for smartwatch-based sound awareness systems will help further research and development in this area, ultimately improving the quality of life for DHH individuals.
Users
5 DHH participants using Assistive Listening Devices (ALD)
The five participants in this study were adults with ages ranging from 21 to 55 years old. Two participants identified as d/Deaf, two as hard of hearing, and one participant was deaf in their right ear. Two participants used hearing aids, and one participant used cochlear implants. All participants were from the Rochester area and had diverse backgrounds in terms of education and employment.
Semi-structured Zoom interviews with live captioning
The intention behind the interviews was to understand the unique needs and perspectives of the DHH community and to gather data that could inform the design of effective and usable sound awareness systems. Live captioning was used during the interviews to ensure effective communication, and the data collected was expected to provide insights into the design of smartwatch-based sound awareness systems that would enhance the quality of life for DHH individuals.
In-Vivo and Axial Coding
We used In-Vivo coding to identify key phrases and words used by participants, and Axial coding to categorize and organize data into themes. In-Vivo coding allowed analysis in participants' own words and Axial coding identified common themes. These techniques are commonly used in qualitative research and allowed for a thorough analysis of the data.
Solution
"I have difficulty figuring out where the sound is coming from."
Use motion to convey the direction of sound and intensity
P1 mentioned that they have difficulty locating the direction of sound, saying, "I have difficulty figuring out where the sound is coming from." P2 agreed, saying, "I feel like sometimes I can't tell if it's coming from the left or the right." Therefore, the system should provide information about the direction of the sound.
"I feel like if people had the ability to customize what sounds they want to appear on their smartwatch, that would be more beneficial."
Make sure you provide a way customize most elements of the UI
P2 said, "I feel like if people had the ability to customize what sounds they want to appear on their smartwatch, that would be more beneficial." This suggests the importance of providing users with the ability to customize the user interface. Therefore, the system should allow users to choose which sounds to prioritize and display.
"I want to be able to filter out certain sounds, like alarms."
Provide the ability to filter out sounds based on algorithm confidence
P2 mentioned that they want the ability to filter the kind of sound notifications they receive at any given time, saying, "I want to be able to filter out certain sounds, like alarms." P5 also mentioned that they would like a system that could notify custom sounds, like someone calling out their name or a particular alarm in their house. Therefore, the system should provide the ability to select specific sounds to detect.
"And also, if it were easier to dismiss the notification…”
Provide an easy way to dismiss and mute the notification
P5 described the current system as "time-consuming" and suggested that making it easier to dismiss notifications would be beneficial: "And also, if it were easier to dismiss the notification, you could say 'Okay, turn off the notification for half an hour or an hour' and do that in the watch or app itself. That would be nice as well." The participants prefer their notifications to be concise, i.e. glanceable, and easy to dismiss. Therefore, ensure that the notification fits on one screen and has a dismiss and a mute button.
Handling Unknown Sounds
Maintain a history of sounds that the user can access
As P5 stated, "maybe you could just let me know in the background. That would be nice. Background kind of report that I can access in real time, maybe. I could see, Okay, these are the sound that is heard, but it's not reporting to me actively." This suggests that users may value having access to information about unknown sounds without being actively notified. Therefore, the system should provide a list of identified sounds, with unknown sounds tagged as "Maybe", and display a sound report to provide more information.
"I have difficulty understanding what people are saying in a noisy environment”
Provide captions in noisy environments and group settings
P2 said, "I have difficulty understanding what people are saying in a noisy environment; that's really a much bigger issue." This feature would enable DHH individuals to better engage with conversations, enhancing their communication experience and reducing feelings of isolation. Therefore, the system should provide live captioning with a dismiss and mute button.
Connecting Assistive Listening Devices
Users should be able to control their ALDs and the programs on the ALDs should match the modes in the app
P1, P2, and P5 wanted a sound awareness system that could also control ALDs. The biggest pain point for ALDs is their batteries running out without warning. We can fix that by notifying users right before their batteries are about to run out. Most ALDs have different programs that provide various sound filters. The modes in the sound awareness system should synchronize with the programs to provide more control for the users. Therefore, the system should display the user's ALD information, including battery status, volume control, and power control. It should also display a notification when the battery in ALDs is running out.
Prototype
success measures
Usability Testing
We need to conduct usability testing with DHH users while they are using the app. We propose a Wizard-of-Oz study that will check the users response in 3 situations like a bus stop, cafe, and build lobby.
how did i change?
Through my graduate capstone project SoundSense, I gained valuable experience in designing technology for the Deaf and Hard of Hearing (DHH) community. Despite challenges in recruiting participants and communicating with DHH individuals during interviews, I was able to complete the project successfully and developed a comprehensive solution that addressed the lack of effective sound awareness technology for DHH individuals.
Through the project, I gained insights into the experiences and preferences of the DHH community regarding sound awareness technologies for smartwatches. I also discovered the sound programs of hearing aids, which I found fascinating. Overall, the project allowed me to grow in my understanding of the challenges and opportunities in designing technology for the DHH community. It inspired me to continue prioritizing inclusivity and accessibility in my work as a designer.
SoundSense: Smartwatch Sound Awareness App for D/deaf and Hard of Hearing Users
Tags
Accessibility, DHH, Smartwatch, Sound Awareness, Capstone Project
Client
Capstone Project
Completed
2023
Context
Enhance the quality of life for D/deaf and hard of hearing (DHH) individuals by providing them with an accessible and customizable way to detect and interpret sounds in their environment
Why do we need sound awareness systems?
DHH individuals often struggle to detect and interpret sounds in their environment, making it difficult for them to navigate their daily lives and engage with their surroundings. The lack of effective sound awareness technology was a major problem for those who were constantly struggling to detect and interpret sounds in their environment.
What are sound awareness systems?
Sound awareness systems are devices or software that help individuals detect and interpret sounds in their environment. The goal of these systems is to enhance the quality of life for DHH individuals by providing them with an accessible way to detect and interpret sounds.
This project was my graduate capstone project with RIT, NY.
Problem Statement
Provide design guidelines and an example for smartwatch based sound awareness system
Previous studies have explored sound awareness technology for this community, with key findings including the importance of personalization and user-centered design. However, there are still limitations in current technology that must be addressed to provide effective and usable sound awareness systems. Therefore, we believe that providing design guidelines and an example for smartwatch-based sound awareness systems will help further research and development in this area, ultimately improving the quality of life for DHH individuals.
Users
5 DHH participants using Assistive Listening Devices (ALD)
The five participants in this study were adults with ages ranging from 21 to 55 years old. Two participants identified as d/Deaf, two as hard of hearing, and one participant was deaf in their right ear. Two participants used hearing aids, and one participant used cochlear implants. All participants were from the Rochester area and had diverse backgrounds in terms of education and employment.
Semi-structured Zoom interviews with live captioning
The intention behind the interviews was to understand the unique needs and perspectives of the DHH community and to gather data that could inform the design of effective and usable sound awareness systems. Live captioning was used during the interviews to ensure effective communication, and the data collected was expected to provide insights into the design of smartwatch-based sound awareness systems that would enhance the quality of life for DHH individuals.
In-Vivo and Axial Coding
We used In-Vivo coding to identify key phrases and words used by participants, and Axial coding to categorize and organize data into themes. In-Vivo coding allowed analysis in participants' own words and Axial coding identified common themes. These techniques are commonly used in qualitative research and allowed for a thorough analysis of the data.
Solution
"I have difficulty figuring out where the sound is coming from."
Use motion to convey the direction of sound and intensity
P1 mentioned that they have difficulty locating the direction of sound, saying, "I have difficulty figuring out where the sound is coming from." P2 agreed, saying, "I feel like sometimes I can't tell if it's coming from the left or the right." Therefore, the system should provide information about the direction of the sound.
"I feel like if people had the ability to customize what sounds they want to appear on their smartwatch, that would be more beneficial."
Make sure you provide a way customize most elements of the UI
P2 said, "I feel like if people had the ability to customize what sounds they want to appear on their smartwatch, that would be more beneficial." This suggests the importance of providing users with the ability to customize the user interface. Therefore, the system should allow users to choose which sounds to prioritize and display.
"I want to be able to filter out certain sounds, like alarms."
Provide the ability to filter out sounds based on algorithm confidence
P2 mentioned that they want the ability to filter the kind of sound notifications they receive at any given time, saying, "I want to be able to filter out certain sounds, like alarms." P5 also mentioned that they would like a system that could notify custom sounds, like someone calling out their name or a particular alarm in their house. Therefore, the system should provide the ability to select specific sounds to detect.
"And also, if it were easier to dismiss the notification…”
Provide an easy way to dismiss and mute the notification
P5 described the current system as "time-consuming" and suggested that making it easier to dismiss notifications would be beneficial: "And also, if it were easier to dismiss the notification, you could say 'Okay, turn off the notification for half an hour or an hour' and do that in the watch or app itself. That would be nice as well." The participants prefer their notifications to be concise, i.e. glanceable, and easy to dismiss. Therefore, ensure that the notification fits on one screen and has a dismiss and a mute button.
Handling Unknown Sounds
Maintain a history of sounds that the user can access
As P5 stated, "maybe you could just let me know in the background. That would be nice. Background kind of report that I can access in real time, maybe. I could see, Okay, these are the sound that is heard, but it's not reporting to me actively." This suggests that users may value having access to information about unknown sounds without being actively notified. Therefore, the system should provide a list of identified sounds, with unknown sounds tagged as "Maybe", and display a sound report to provide more information.
"I have difficulty understanding what people are saying in a noisy environment”
Provide captions in noisy environments and group settings
P2 said, "I have difficulty understanding what people are saying in a noisy environment; that's really a much bigger issue." This feature would enable DHH individuals to better engage with conversations, enhancing their communication experience and reducing feelings of isolation. Therefore, the system should provide live captioning with a dismiss and mute button.
Connecting Assistive Listening Devices
Users should be able to control their ALDs and the programs on the ALDs should match the modes in the app
P1, P2, and P5 wanted a sound awareness system that could also control ALDs. The biggest pain point for ALDs is their batteries running out without warning. We can fix that by notifying users right before their batteries are about to run out. Most ALDs have different programs that provide various sound filters. The modes in the sound awareness system should synchronize with the programs to provide more control for the users. Therefore, the system should display the user's ALD information, including battery status, volume control, and power control. It should also display a notification when the battery in ALDs is running out.
Prototype
success measures
Usability Testing
We need to conduct usability testing with DHH users while they are using the app. We propose a Wizard-of-Oz study that will check the users response in 3 situations like a bus stop, cafe, and build lobby.
how did i change?
Through my graduate capstone project SoundSense, I gained valuable experience in designing technology for the Deaf and Hard of Hearing (DHH) community. Despite challenges in recruiting participants and communicating with DHH individuals during interviews, I was able to complete the project successfully and developed a comprehensive solution that addressed the lack of effective sound awareness technology for DHH individuals.
Through the project, I gained insights into the experiences and preferences of the DHH community regarding sound awareness technologies for smartwatches. I also discovered the sound programs of hearing aids, which I found fascinating. Overall, the project allowed me to grow in my understanding of the challenges and opportunities in designing technology for the DHH community. It inspired me to continue prioritizing inclusivity and accessibility in my work as a designer.
Chirag Ghube
Explore
UX Designer & Web Developer
SoundSense: Smartwatch Sound Awareness App for D/deaf and Hard of Hearing Users
Tags
Accessibility, DHH, Smartwatch, Sound Awareness, Capstone Project
Client
Capstone Project
Completed
2023
Context
Enhance the quality of life for D/deaf and hard of hearing (DHH) individuals by providing them with an accessible and customizable way to detect and interpret sounds in their environment
Why do we need sound awareness systems?
DHH individuals often struggle to detect and interpret sounds in their environment, making it difficult for them to navigate their daily lives and engage with their surroundings. The lack of effective sound awareness technology was a major problem for those who were constantly struggling to detect and interpret sounds in their environment.
What are sound awareness systems?
Sound awareness systems are devices or software that help individuals detect and interpret sounds in their environment. The goal of these systems is to enhance the quality of life for DHH individuals by providing them with an accessible way to detect and interpret sounds.
This project was my graduate capstone project with RIT, NY.
Problem Statement
Provide design guidelines and an example for smartwatch based sound awareness system
Previous studies have explored sound awareness technology for this community, with key findings including the importance of personalization and user-centered design. However, there are still limitations in current technology that must be addressed to provide effective and usable sound awareness systems. Therefore, we believe that providing design guidelines and an example for smartwatch-based sound awareness systems will help further research and development in this area, ultimately improving the quality of life for DHH individuals.
Users
5 DHH participants using Assistive Listening Devices (ALD)
The five participants in this study were adults with ages ranging from 21 to 55 years old. Two participants identified as d/Deaf, two as hard of hearing, and one participant was deaf in their right ear. Two participants used hearing aids, and one participant used cochlear implants. All participants were from the Rochester area and had diverse backgrounds in terms of education and employment.
Semi-structured Zoom interviews with live captioning
The intention behind the interviews was to understand the unique needs and perspectives of the DHH community and to gather data that could inform the design of effective and usable sound awareness systems. Live captioning was used during the interviews to ensure effective communication, and the data collected was expected to provide insights into the design of smartwatch-based sound awareness systems that would enhance the quality of life for DHH individuals.
In-Vivo and Axial Coding
We used In-Vivo coding to identify key phrases and words used by participants, and Axial coding to categorize and organize data into themes. In-Vivo coding allowed analysis in participants' own words and Axial coding identified common themes. These techniques are commonly used in qualitative research and allowed for a thorough analysis of the data.
Solution
"I have difficulty figuring out where the sound is coming from."
Use motion to convey the direction of sound and intensity
P1 mentioned that they have difficulty locating the direction of sound, saying, "I have difficulty figuring out where the sound is coming from." P2 agreed, saying, "I feel like sometimes I can't tell if it's coming from the left or the right." Therefore, the system should provide information about the direction of the sound.
"I feel like if people had the ability to customize what sounds they want to appear on their smartwatch, that would be more beneficial."
Make sure you provide a way customize most elements of the UI
P2 said, "I feel like if people had the ability to customize what sounds they want to appear on their smartwatch, that would be more beneficial." This suggests the importance of providing users with the ability to customize the user interface. Therefore, the system should allow users to choose which sounds to prioritize and display.
"I want to be able to filter out certain sounds, like alarms."
Provide the ability to filter out sounds based on algorithm confidence
P2 mentioned that they want the ability to filter the kind of sound notifications they receive at any given time, saying, "I want to be able to filter out certain sounds, like alarms." P5 also mentioned that they would like a system that could notify custom sounds, like someone calling out their name or a particular alarm in their house. Therefore, the system should provide the ability to select specific sounds to detect.
"And also, if it were easier to dismiss the notification…”
Provide an easy way to dismiss and mute the notification
P5 described the current system as "time-consuming" and suggested that making it easier to dismiss notifications would be beneficial: "And also, if it were easier to dismiss the notification, you could say 'Okay, turn off the notification for half an hour or an hour' and do that in the watch or app itself. That would be nice as well." The participants prefer their notifications to be concise, i.e. glanceable, and easy to dismiss. Therefore, ensure that the notification fits on one screen and has a dismiss and a mute button.
Handling Unknown Sounds
Maintain a history of sounds that the user can access
As P5 stated, "maybe you could just let me know in the background. That would be nice. Background kind of report that I can access in real time, maybe. I could see, Okay, these are the sound that is heard, but it's not reporting to me actively." This suggests that users may value having access to information about unknown sounds without being actively notified. Therefore, the system should provide a list of identified sounds, with unknown sounds tagged as "Maybe", and display a sound report to provide more information.
"I have difficulty understanding what people are saying in a noisy environment”
Provide captions in noisy environments and group settings
P2 said, "I have difficulty understanding what people are saying in a noisy environment; that's really a much bigger issue." This feature would enable DHH individuals to better engage with conversations, enhancing their communication experience and reducing feelings of isolation. Therefore, the system should provide live captioning with a dismiss and mute button.
Connecting Assistive Listening Devices
Users should be able to control their ALDs and the programs on the ALDs should match the modes in the app
P1, P2, and P5 wanted a sound awareness system that could also control ALDs. The biggest pain point for ALDs is their batteries running out without warning. We can fix that by notifying users right before their batteries are about to run out. Most ALDs have different programs that provide various sound filters. The modes in the sound awareness system should synchronize with the programs to provide more control for the users. Therefore, the system should display the user's ALD information, including battery status, volume control, and power control. It should also display a notification when the battery in ALDs is running out.
Prototype
success measures
Usability Testing
We need to conduct usability testing with DHH users while they are using the app. We propose a Wizard-of-Oz study that will check the users response in 3 situations like a bus stop, cafe, and build lobby.
how did i change?
Through my graduate capstone project SoundSense, I gained valuable experience in designing technology for the Deaf and Hard of Hearing (DHH) community. Despite challenges in recruiting participants and communicating with DHH individuals during interviews, I was able to complete the project successfully and developed a comprehensive solution that addressed the lack of effective sound awareness technology for DHH individuals.
Through the project, I gained insights into the experiences and preferences of the DHH community regarding sound awareness technologies for smartwatches. I also discovered the sound programs of hearing aids, which I found fascinating. Overall, the project allowed me to grow in my understanding of the challenges and opportunities in designing technology for the DHH community. It inspired me to continue prioritizing inclusivity and accessibility in my work as a designer.