2022 SNAAP Technical Report

II.A. Questionnaire Development

II.B Substantive Expansion

II.C Pilot Test Administration

III.A Population File

III.B Response Reminders and Incentives

III.C Response Rate

III.D Data Considerations and Preparations

IV.A CIP Codes

IV.B Recruitment of Institutional Participants

IV.C Weighting Procedures

Appendix A: Annotated 2022 SNAAP Questionnaire

Appendix B: Frequency Report of Unweighted 2022 SNAAP Data

Appendix C: Resources for Participating Institutions (Bulletin & FAQ)

Appendix D: SNAAP 2022 Sample Weighting Procedures

In spring 2021, a review of past SNAAP survey administrations’ questionnaires and data quality was undertaken with attention paid to means for streamlining, reducing cognitive burden, reducing breakoffs, and updating the overall questionnaire. Key topical areas were identified for continuation and for expansion. Key areas for continuation included, but were not exclusive to, alumni work and employment, as well as satisfaction with postsecondary education institutions. Key areas for expansion included, but were not exclusive to, socio-demographic characteristics, sense of belonging, implications of the COVID-19 pandemic for training, employment, and ways of work, and paying for postsecondary arts, design, and related educations. Once an initial draft was developed, critical feedback was sought from an array of stakeholders. Additionally, a further revised draft questionnaire was made publicly available for input and comments on the SNAAP website in fall 2021 and spring 2022.

In fall 2021, four focus groups and three interviews were held with higher education administrators, faculty, and additional stakeholders to discuss information needs related to the lives and careers of arts, design, and related alumni under University of Illinois Urbana-Champaign IRB. The purpose of these conversations was twofold: (a) to garner insights on pressing information needs within postsecondary arts and design fields as they relate to understanding and improving the careers and lives of graduates of related degree programs and (b) to generate feedback on the draft 2022 SNAAP questionnaire. The results of the former are synthesized and shared in the report, Data, Pressing Needs, and Biggest Challenges: Insights from the Field (Novak-Leonard, Dempster, Scotto Adams, & Walters, 2022).

A series of questions relevant to insights regarding alumni perspectives on their sense of belonging while at their institution were cognitively tested with 33 alumni of arts, design, and media production undergraduate degrees.

The main objective of the cognitive testing was to probe the comprehension and wording of the questions, whether the questions were clear, and the interviewees’ thoughts on them. However, participants were at liberty to answer the questions as they deemed fit.

The interviewees were recruited from amongst recent alumni, defined as those who completed their degrees within the prior five years of receiving the invitation to participate in the interview. Interviewees stemmed from a range of disciplinary degree foci; institutional types, inclusive of a R1 research university, a specialized arts and design school, and minority-serving postsecondary institutions. These same interviewees also answered questions about sense of belonging, which are part of a larger qualitative study being led by Dr. Jennifer Novak-Leonard. Interviewees were given a $40 Amazon e‑gift card for participating in the whole interview, paid for with funds supplied by the University of Illinois Urbana-Champaign.

The interviews were conducted via Zoom, and each was 30 – 75 minutes in length, with most interviews concluding within 60 minutes. During the interviews, the interviewer read out the questions to interviewees and placed them in the chat. Interviews were conducted by Dr. Jennifer Novak-Leonard, Emma Walters, and Dr. Shanita Bigelow, under University of Illinois Urbana-Champaign IRB.

A specific aim of the 2022 SNAAP survey administration was to capture more nuanced data on how individual alumni identify themselves, and on their social identities and socioeconomic circumstances in effort to garner more granular insights on the varied experiences of alumni within their postsecondary studies and training and experiences since. Reviews of current, albeit evolving, best practices for inclusively collecting information on race and ethnicity, gender identity, sexual orientation 2, and personal and household income were conducted, and comparable measures adapted for use within the 2022 SNAAP questionnaire. These measures are included in the latter questionnaire section entitled ​“Identity & Socio-Demographic Questions.”

2. For example, see Morgan et al. (2020).

A key aim of the 2022 SNAAP survey administration was to document arts and design alumni’s experiences with sense of belonging during their postsecondary arts and design education, as well as within their employment if they were in the workforce at the time of 2022 SNAAP survey administration; these measures are included in Section A and Section C, respectively, of the 2022 SNAAP questionnaire. Sense of belonging is recognized as a key indicator of student well-being within higher education and as a predictor of student outcomes (Bentrim & Henning, 2022).

An aim of the 2022 SNAAP survey was to document arts and design alumni’s experiences during and after the COVID-19 pandemic, beginning in March 2020. The items developed for Section D of the questionnaire address pandemic-related experiences with work, artistic and creative practice, professional networks, skill gaps, and new skill acquisition. Attention to these topics is essential for establishing how arts alumni fared during the pandemic, including potential challenges they faced and areas where they may have upskilled or shifted focus to new artistic or non-artistic skills to respond to the changing world of work.

To develop survey questions to meet these research aims, Dr. Rachel Skaggs and Dr. Elizabeth Cooksey at The Ohio State University, along with research assistants MollyJo Burke and Erin J. Hoppe, conducted 66 interviews with arts alumni whose artistic work, educational experiences, and personal demographics closely resemble past SNAAP survey samples (read more in Skaggs, 2023; Skaggs, Hoppe, & Burke, 2022). Interviews were conducted from November 2020 – March 2021 and focused primarily on the impact of the pandemic to that point in time on arts graduates’ careers, personal lives, and their future. This research team also conducted informational interviews with eight administrative leaders in higher education in the arts. These deans, provosts, and presidents of institutions identified pressing challenges and sites of resilience and innovation for their institutions, students, and alumni that were instructive in designing Section D of the 2022 SNAAP questionnaire.

Appendix A contains the annotated 2022 SNAAP questionnaire with reference to influences toward the development of these questions. Questions and items that are not explicitly annotated were developed based on findings from the previously described interviews and from considerations of how to amend existing SNAAP questions from prior survey administration to be relevant and responsive to the aims of the 2022 SNAAP survey and related research.

The 2022 SNAAP survey also aimed to capture more nuanced information about how alumni had financed their educations and to gauge related debt. Relatedly, the SNAAP survey sought to better understand career-related motivations for those who pursued graduate degrees in content of education financing. The SNAAP survey is administered to alumni of arts, design, and adjacent programs, and therefore the SNAAP dataset itself does not contain data on other degrees in other fields, which limits the types of comparative analyses possible between alumni of arts and design programs and other fields, such as science, engineering, etc. In effort to enable some comparative contextualization with alumni of other degree fields, the questions used in this section of the 2022 SNAAP questionnaire are adopted from the 2019 National Survey of College Graduates managed by the National Science Foundation, which largely focuses on alumni of science and engineering programs.3

3. For more information about the National Survey of College Graduates, see: https://www.nsf.gov/statistics/srvygrads-legacy/#

In the summer of 2022, a pilot test was conducted, with a sample of alumni from two institutions planning to participate in the fall 2022 administration. Each institution was asked to provide contact information for no more than 1,000 of their alumni. Although both participating institutions considered providing incentives to their alumni who completed the survey, neither institution ultimately decided to provide incentives for the pilot study. One institution requested to include a recent sample of alumni (2017 graduation year or sooner), and the other included a sample that was representative across graduation years. The contacts at these institutions used the population file instruction guide and worked with the Director of Research Services to generate the alumni population file. In each alumni file, schools were required to include necessary information, such as the name, email address(es), graduation year, area of study, department, college/school, and CIP code of arts and design alumni. Schools were also given the option to include the alum’s street address(es), GPA, gender, alumni ID, and any additional information they would like to be included in the final dataset that was not asked in the survey. Each school submitted a file for 1,000 alumni to SNAAP staff and the Indiana University Center for Survey Research (CSR; SNAAP’s survey programming and administration service partner) using a secure OneDrive folder, resulting in a total pilot pool of 2,000 alumni to recruit from.

Once the files were reviewed by SNAAP and CSR, they were shared with AlumniSync, the alumni search firm, via a secure folder. AlumniSync was responsible for validating the email addresses on file and providing any updated email addresses if detected. Of the 2,000 alumni included in the file, there were 1,657 (83%) with at least a primary email address on file. Of those 1,657 addresses, AlumniSync was able to validate 1,520 (92%); inother words, 1,520 email addresses were valid and safe to contact — that is 76% of the total 2,000 contacts submitted. Conversely, there were 137 email addresses on file that came back as invalid or unsafe to contact. Of those 137, 35 had a secondary email address on file, and AlumniSync was able to find 69 email addresses based on alumni contact information. In total, the search resulted in 1,745 valid email addresses (83% of the total 2,000; 105% of the total 1,657 with an email address on file) for CSR to send recruitment messages to, resulting in a recruitment pool of 1,745 alumni.

SNAAP worked closely with CSR to develop a set of five recruitment messages (one initial invitation, and four reminders), to be sent out to the recruitment pool. Given the short timeline of pilot survey administration of just over two weeks, we decided to remove one of the reminder messages. Once the messages were developed, SNAAP coordinated with our institutional contacts to allow them to customize certain sections of the messages and add their school logos. The contacts were also sent a test version of the first recruitment message via email before survey administration began, so that they could get a sense of what alumni would receive and add any final requests.

The pilot survey launched on June 28th, 2022, and closed on July 15th, 2022. A total of four recruitment messages were sent to alumni:

• Initial Invitation: Tuesday, June 28, 2022
• Reminder 1 Message: Tuesday, July 5, 2022
• Reminder 2 message: Friday, July 8, 2022
• Reminder 3 message: Tuesday, July 12, 2022

In accordance with SNAAP’s IRB protocol, if there was a secondary email address on file, a maximum of two recruitment messages were sent to multiple emails. Additionally, one message was sent in HTML formatting to improve the diversity of text used.

In addition to recruitment strategies, SNAAP encouraged contacts at our participating institutions to promote the SNAAP survey to their alumni via email and/or social media accounts, both before and during pilot survey administration. Schools were encouraged to send out a ​“pre-survey announcement” before the survey launch, as well as promotional notices throughout administration, to raise the importance of survey participation, and to notify alumni that if they did not receive an invitation yet, they would have the chance to participate in the fall. The recruitment messages would be updated slightly for the fall survey launch to match the fall administration dates.

In total, we received 68 complete responses and 26 partial or break-off responses, for a total of 94 responses. The number of valid contacts AlumniSync identified was 1,520. The adjusted response rate across institutions was 5.7% ((the number of completions + partial completions) / (the total number of alumni — (those with no invitation sent + those with an invitation returned undelivered)).

Considering lower-than-expected response rates, SNAAP has explored several potential contributing factors.

• Recent research suggests that survey response rates are on a decline and survey fatigue is on an incline, particularly since the COVID-19 pandemic (de Koning et al., 2021). Due to a recent increase in web-based surveys, increased spam messages, and other reasons, web-based surveys have been linked to lower response rates than other methods (Daikeler, Bošnjak, & Lozar Manfreda, 2019).
• The short, two-week window of the pilot survey administration provided less time for alumni to participate than the full survey does.
• Our alumni search firm this year was able to find a higher number of alumni emails than we normally have access to, making for a lower overall response rate.

Given these factors, SNAAP’s pilot response rates are aligned with other recent alumni surveys. For instance, an alumni survey conducted by Fordham University in 2019 yielded a 5% response rate, and one by Cornell University in 2017 yielded a 7% rate.

Once the pilot data was returned, the data was cleaned and thoroughly checked. Alumni comments within the survey were reviewed to determine whether changes to the survey were warranted prior to the fall administration. Based on alumni responses and experiences with the survey, the research team decided on several minor changes to implement for the larger survey administration:

1. We decided to include fine arts degrees (BFAs and MFAs) in response options that listed degrees. For instance, in the pilot survey, the response option at the bachelor’s degree level included BA, BS, BM, etc.; we added BFA as an example. We made this change because many participating alumni are reporting BFA and MFA degrees.
2. We decided to edit the response option ​“j” to curjob_ series (items referring to non-arts jobs). Rather than being specific to social services, we broadened the category to ​“social and other services” to include food service and travel industry jobs. We made this change because several participants wrote in other types of service industry responses.
3. We decided to paraphrase the prompt for the ​‘wh_whynot_’ series (items assessing why a participant is not working for pay or profit) to improve readability. This involved a slight adjustment from, ​“What are your reasons for not currently working for pay or profit?” to, ​“Please indicate why you are not currently working for pay or profit.” We made this change because an alum indicated in the open-ended response that they did not understand the question.
4. We updated the response options to the survey item assessing gender. We made this change because several participants used the ​‘other’ text box to indicate that they were nonbinary, which was not an option originally included in the item. We changed the response options from ​‘female’, ​‘male’, and ​‘a gender identity that is not listed (e.g., gender fluid, two- spirit, transgender male),’ to the following: ​“Woman,” ​“Man,” ​“Gender identity that is not listed (e.g., nonbinary, transgender, gender-fluid, two-spirit), please share: [TEXT BOX, 100 ch. limit],” ​“Prefer not to answer.”

All the above revisions were implemented to improve readability and inclusion in the 2022 survey.

The process for collecting alumni data from institutions for the fall survey administration was like that of the pilot survey administration. Of the 120 participating institutions, several registered separately for smaller arts and design schools/colleges within the university, resulting in a total of 128 population files. SNAAP worked closely with the Center for Survey Research to assist schools in providing alumni data and to review population files closely for inconsistencies. CSR was able to program a set of initial checks within the upload feature on the SNAAP interface. This allowed the feature to catch large, systematic errors within the files before they were uploaded. One important part of this process was ensuring that the required fields in the population file were complete. Within the population file instructions, SNAAP included thresholds of necessary data for each category. For instance, SNAAP required that each school provide the first and last name of 100% of alumni, but the email address(es) for at least 25% of alumni — given the ability to identify new email addresses with our third-party search firm. The upload feature would not allow institutional contacts to upload the population file if less than 100% of alumni had a first and last name listed, or if less than 25% had an email address listed.

Due to the complex nature of the population file, many schools reached out with questions, which SNAAP and CSR documented. The largest challenges schools faced throughout the process were:

1. Locating and providing accurate CIP (Classification of Instructional Programs) codes for alumni.
   1. Over half of the participating institutions originally provided at least one CIP code for alumni that was outside of the sampling frame. In most cases, the alum had an arts or design major on file, but the CIP code itself was not within the identified frame. Alumni who had an arts or design major on file were included. In a handful of cases, there were more systematic inconsistencies — such that the school provided many alumni with non-arts or design majors and non-arts or design CIPs. In those cases, we reached out to schools to confirm that these alumni were indeed not within the arts or design, and flagged ineligible alumni not to be recruited.
2. Inconsistencies in alumni degree information.
   1. A portion of schools provided contrasting information when referring to alumni degrees. For instance, the reported degree level (e.g., a doctoral degree, a master’s degree, a bachelor’s degree) did not match the reported degree level detail (e.g., PhD, MFA, BA), resulting in an alum’s degree level of a master’s degree but degree level detail of a BA. Another common inconsistency occurred in reporting the highest arts degree. Schools were asked to report the highest arts degree under degree level and degree level detail, and were asked to report additional (lower) degrees under arts degree 2 and arts degree 3. However, at times schools reported higher arts degrees in arts degree 2 or arts degree 3 rather than in the degree level column. Each of these inconsistencies were corrected through direct communication with schools. SNAAP reached out to institutional contacts and clarified both the correct degree and the highest arts degree received.
3. Including duplicate cases of alumni in one population file, or duplicate cases of alumni across multiple population files.
   1. A portion of schools included duplicate cases of alumni in their institution’s population file; within-file duplicate cases were indicated by multiple cases of the same name and email address or multiple cases of the same email address with different names. SNAAP communicated directly with schools to clarify when duplicate records were included. In most cases of the same email address across multiple names, multiple alumni (family members, spouses) were sharing an email address according to school records. Most cases of the same name and email address were typos within school records. Regarding alumni who were included across multiple population files, SNAAP and CSR developed guidelines for alumni recruitment. To avoid sending the same alumni several survey emails relevant to multiple schools, we decided to contact alumni through contact information provided by the institution at which they received their highest arts degree. If multiple schools reported arts degrees at the same degree level for the same alumni, we used their contact information from the school associated with the most recent graduation year.

Throughout the survey administration, SNAAP received requests from institutions to add alumni to the recruitment list who were not originally included in their population files. In many cases, these alumni saw promotional messages about the survey on social media or via email and requested to be added, or these alumni were cases that schools were otherwise unable to include in their original file. SNAAP and CSR established a system through which schools could upload a condensed version of the population file for only those cases they needed to add by a given cutoff date during the survey administration.

Similar to the pilot administration, SNAAP engaged a third-party alumni contact search firm, AlumniSync, for the fall survey administration. Two institutions opted out of this process due to data sharing concerns. SNAAP shared a total of 1,044,084 alumni records with AlumniSync, but of those, 796,959 records had an email address on file. AlumniSync was able to verify 88% of those alumni’s email addresses as ​“safe” to send. In addition to verifying email addresses, AlumniSync found an additional 151,535 new email addresses for alumni who only had a ‘.edu’ address (which we know have low response rates from the pilot study), and for alumni with a missing or invalid email address. At the end of AlumniSync’s search, a total of 823,791 alumni were considered to have valid email addresses on file. However, it is important to note that a number of those addresses considered valid bounced back during the recruitment stage (see below); the response rate was adjusted accordingly. After the first survey invitations were emailed to all alumni with a valid email address on file, 42,884 email addresses initially bounced back.

AlumniSync ran an additional validation check and was able to replace 13,999 of those emails with new emails considered safe to send.

Throughout the survey administration, alumni received a series of recruitment messages via email. These messages included the link to the survey. Below is the full schedule of messages for the standard administration:

• Invitation: October 11 — 14
• Reminder 1: October 18 — 21
• Reminder 2: October 25 — 28
• Reminder 3: November 1 — 4
• Reminder 4: November 8 — 11
• Additional/final Reminder: November 15 — 22

Both CSR and SNAAP closely monitored participant response rates and engagement throughout the administration. CSR programmed daily response rate updates that SNAAP could use to track participation. The response rates following the initial invitation email were not quite as high as SNAAP and CSR expected, and given the historical knowledge that the first invitations tend to bring the highest number of responses than any other message, the team immediately brainstormed and implemented the following strategies for boosting response rates for the remainder of the administration:

• SNAAP developed and sent out a bulletin to institutional contacts to encourage them to promote the survey to alumni. The bulletin went out in between reminders 1 and 2, in order to prevent survey messages from getting lost in alumni’s inbox. The bulletin included text that schools could copy and paste into their social media accounts.
• SNAAP established a system with CSR that allowed for schools to text the survey link to their alumni if they had the permission and interest to do so. CSR was able to send the school a list of alumni who have not yet taken the survey and schools could text alumni directly, with guidance from an FAQ SNAAP developed.
• To increase reach, CSR programmed reminder 3 emails going out to “.edu” addresses to come from the “.edu” SNAAP email (snaapsrv@indiana. edu), and the reminder 3 emails going out to “.org” and “.com” addresses to come from the “.org” SNAAP email.
• For reminder 3, CSR conducted split-testing to understand whether a change in subject line impacts response rates. The original subject line was: Don’t miss out! Take part in a national arts alumni survey – FULL INSTITUTION NAME wants to hear from you! The revised subject line was: Don’t miss out – SHORT INSTITUTION NAME wants to hear from you
• The first batch of reminder 3 messages were scheduled to go out using plain text, rather than HTML. However, CSR noticed a drop in responses, consistent with the pilot, when using plain text. Because our IRB status (exempt) allowed for changes to the recruitment strategy, SNAAP decided to send the rest of the reminder 3 messages out in HTML formatting.
• As noted above, SNAAP allowed institutions to upload addendums to their population files to include any alumni who were not originally listed in the recruitment list.

Approximately 60 participating schools offered incentives to their alumni to take the 2022 SNAAP survey. This is a substantial increase from previous survey years, when approximately 10 schools would offer an incentive. Incentives this year ranged from institutional swag offerings to drawings for iPods, Amazon gift cards, and event tickets.

In addition to school incentive offerings, SNAAP offered direct survey incentives of $30 e‑gift cards to alumni respondents of institutions typically underrepresented in the SNAAP survey, including 1 tribal college, 3 HBCUs, and 7 community colleges.