Student Attitudes Toward Digital Badges for Instructional Design Competency-Based Education

: This case study examines the self-reported attitudes of graduate students in an online instructional design master’s program regarding its use of digital badges to track and record students’ required professional competencies. The study sought to evaluate student attitudes towards the program and its use of badges. All enrolled students, approximately 300, were asked to complete a course improvement survey that included the Attitudinal Learning Inventory which measured their perceptions of how their attitudes changed toward the use of digital badges for competency-based education and what aspect of the course most impacted their attitudes. 211 students responded to the course improvement survey. 14 students, ranging from those who had completed the program to those who had only completed a single semester were also interviewed to gather richer data detailing their positive and negative experiences with demonstrating competence through the digital badge process. Results indicated that the students had positive attitudes regarding the use of digital badges across all components of their attitudes-cognitive, affective, behavioral, and social. Participants noted how the badges helped their learning and motivation while also identifying aspects of the program that worked best for them and areas which could be improved. Implications are that digital badges for tracking competencies can improve learning and motivation, but the implementation benefits from significant planning and process revision.


Introduction
Competency-based education (CBE) is a pedagogical approach that focuses on mastering learning outcomes; and students' progress based on demonstrating mastery of required skills and knowledge captured by predefined competencies (Henri et al., 2017).As opposed to traditional seat-time and grade-based approaches to education, CBE focuses on the performance of instructional goals, establishes a standard of acceptable performance rather than comparing learners, informs learners and other stakeholders what is expected upon completion, expresses learning through assessable behaviors, and reflects the application of what has been learned (Henri et al., 2017).Higher education has seen a trend from a knowledge-based focus to CBE in recent decades (Samuelowicz & Bain, 2001), moving from what conceptually needs to be taught and learned to what needs to be taught, why, and how it should be applied to solve complex problems (Hoogveld et al., 2005).To successfully make the transition from knowledge-based to CBE, the role of instructional design (ID) is key in order to effectively analyze the required competencies, making ID a core competency for instructors and trainers implementing CBE approaches (Hoogveld et al., 2005;Tennyson, 2001).
Digital badges (DB) are increasingly being adopted internationally in higher education (Grant, 2016) and have been utilized to certify competence (Hartnett, 2021).However, there has been limited empirical research on their implementation (Carey & Stefaniak, 2018).This study examines the effectiveness of an online ID master's program that implements CBE, utilizing DB for credentialing completion of professional ID competencies.Effectiveness is studied through the self-reported perceptions of program students over time as they seek to demonstrate professional competence through the DB.

Literature review
This study examines the self-reported attitudes of online ID master's students towards the use of DB to certify their proficiency with specific ID competencies.As current and future IDers, they will be tasked with the use of learning technologies to support the instruction they design, and DB are one option for the assessment and feedback component of the learning process.Therefore, in examining the effectiveness of CBE in their program, it is important to consider their attitudinal learning over time in the program, and how their perspective towards the use of DB for CBE changed as they experienced it as a core element of the ID learning.

DB and attitudinal learning
DB are micro-certification units that provide evidence of knowledge, skill, and experience (Davies et al., 2015;Randall et al., 2013).They outline "incremental learning pathways" that serve as visual representations of specific skills targeted towards achieving larger goals (Gish-Lieberman et al., 2021, p. 5).DB can be defined as "a representation of an accomplishment, interest or affiliation that is visual, available online, and contains metadata including links that help explain the context, meaning, process and result of an activity" (Gibson et al., 2013, p. 404).They are widely recognized as innovative pedagogical tools to enhance learning experiences in various fields (Gamrat et al., 2014;Higashi et al., 2012;Santos et al., 2013;Wallis & Martinez, 2013) and to validate prior learning (Ahn et al., 2014).
Studies have shown DB support self-regulated learning (Cucchiara et al., 2014), increase student interaction or engagement (Chou & He, 2017), increase motivation (Ahn et al., 2014;Glover, 2013), and help recognize informal and formal competencies (Farmer & West, 2016).DB facilitate the goal-setting process (Cheng et al., 2018;McDaniel & Fanfarelli, 2016) and help clarify learning objectives, deliver instructional materials, and learning activities and deliver performance evaluation and feedback (Newby & Cheng, 2020).Kalogiannakis and colleagues' (2021) systematic review of gamification in science education identified DB as one of the most common approaches to gamification.
Gamification utilizes game mechanics and elements to motivate or engage in a non-game context (Kalogiannakis et al., 2021;Zourmpakis et al., 2023).Most associated with self-determination theory, gamification seeks to motivate or engage students to achieve learning outcomes (Kalogiannakis et al., 2021) by promoting their participation in the learning process (Zourmpakis et al., 2022).
DB can be used as a gamified approach to display learning goals and objectives clearly and transparently (Zourmpakis et al., 2022).With gamification's focus on student engagement, including awareness of their own learning, attitudinal learning is particularly relevant as a targeted outcome.Attitudinal learning focuses not just on the cognitive outcomes typically examined in learning but additional components that encompass not just what an individual learns, but how that shapes their feeling and behavior due to that learning (Kamradt & Kamradt, 1999).

DB and attitudinal learning
Attitude is the psychological evaluation a person has about an object, person, or event (Gagne et al., 1992) and is comprised of three interrelated components: cognitive, affective, and behavioral (Kamradt & Kamradt, 1999;Simonson, 1979).The cognitive component refers to the evaluations based on information or knowledge; the affective component refers to evaluations based on feelings or emotions; and the behavioral component refers to actions performed towards the object, person, or event (Kamradt & Kamradt, 1999;Simonson, 1979).Social learning, focusing on learning through interaction with others, is another component that influences attitudes (Watson et al., 2018).

DB and cognitive learning
DB integrate instructional elements and strategies (Ahn et al., 2014) and learners follow a learning trajectory or pathway earning badges as they progress (Higashi et al., 2012).Badges help clarify learning objectives, deliver instructional materials, learning activities and deliver performance evaluation and feedback (Newby & Cheng, 2020).They include the criteria to earn the badge besides providing opportunities to demonstrate learned competencies (Devedžić & Jovanović, 2015;Dyjur & Lindstrom, 2017;Newby et al., 2016).Newby and Cheng (2020) found that badges positively impacted pre-service teachers' perceptions of skill acquisition and improved their learning performance by helping them focus on learning goals, using mastery-oriented personalized feedback.Mastery-oriented feedback incorporated in badges addresses the learning outcome, decreases learning gaps (Besser, 2016), and guides learners towards achieving a skill or knowledge (Fanfarelli et al., 2015).

DB and affective learning
Acting as "powerful incentives" (Anderson et al., 2013), DB can enhance student motivation to learn (Ahn et al., 2014;Denny, 2013;Glover, 2013) and increase perceived learning by providing positive feedback that gives a sense of accomplishment (Newby & Cheng, 2020).Badges can also remove the negative perceptions of assessments that exist in traditional educational systems and reduce levels of performance avoidance goals (Higashi et al., 2012).

DB and behavioral learning
DB can also help identify progress and increase time on task (McDaniel & Fanfarelli, 2016).Students have reported changes in behaviors because of badges such as in time management, number of submissions, and making submissions perfect the first time (Haaranen et al., 2014).Hakulinen and Auvinen (2014) found that most of the students who had higher mastery-intrinsic, mastery-extrinsic, and performance-approach orientation, and lower avoidanceorientation were more motivated to earn badges.

DB and social learning
DB can provide social incentives, and networked support noted Kwon et al. (2015).In an online course, study participants indicated the need to add a social element for sharing badges with friends, to improve the badging system, reported Haaranen et al. (2014).

DB in ePortfolios
In this study, the DB were implemented as part of a professional ePortfolio required for final approval prior to graduation.Like DB, ePortfolios are a pedagogical tool that provide learners with the opportunity to curate artifacts, reflect on their learning, and make connections across experiences (Kuh et al., 2018;Parkes et al., 2013).As both a product and a process, ePortfolios allow students to showcase their learning to a variety of external audiences and deepen their own metacognitive learning (Cheng & Chau, 2013;Munday, 2017;Parkes et al., 2013;Pike et al., 2020;Roberts et al., 2016;Wakimoto et al., 2019;Wuetherick & Dickinson, 2015).Several challenges in integrating ePortfolios into program curriculum consistently appear in the literature.One documented challenge is the time commitment required, by both learners and instructors (Lin, 2008;Slade & Downer, 2020;Wright et al., 2002).Another common challenge is learners' lack of perceived purpose for the portfolio creation (Wright et al., 2002).
This study aims to answer the following research questions: • What are graduate, online ID students' attitudes regarding the use of DB for CBE within the online Learning Design and Technology (LDT) master's program?
• Is there a difference between preferences in learning activities with respect to perceived attitudinal learning gains?
• How do students regard the process of earning DB for CBE?

Research context and course sequence description
This study investigates students' learning experiences within a series of competency earning courses utilizing DB in an online ID master's degree program (see Table 1).Students develop a professional ePortfolio which helps them demonstrate professional competency (on what?) before graduating.Program faculty selected or crafted 16 competencies (each earned through DB) in four different overarching categories (each represented as a supra-badge): 1) professional foundations in ID and technology, 2) design and development, 3) planning and analysis, and 4) evaluation and implementation.Each competency was represented by a list of challenges (tasks) that students needed to successfully complete, have reviewed by peers, and be approved by a course instructor to earn the associated badge.There are 35 challenges in total, and students may revise and resubmit them in future courses after receiving feedback if their instructors reject the submission as not representing evidence of professional competence.Adequate progress in the program demonstrated through both course grades and earned badges is required to persist and graduate from the program.

Data sources
Participants in this study were students enrolled in an online LDT masters at a public, research university in the Midwestern United States.While demographic data was not gathered, students in the program tend to be adults working full time while pursuing their degree, who predominantly, but not entirely, live in the United States.After receiving approval from our institution's ethical research oversight committee, we first use Qualtrics to capture student responses to a course improvement survey.The course improvement survey was embedded in all competency courses as a part of the course assignments at the end of the course for two semesters-Summer 2020 and Fall 2020.Survey participants represented all badge classes; therefore, participants ranged from those in their first badge course to those working on their ePortfolio and finishing the program.There were approximately 150 students enrolled in the competency courses each of the two semesters.Not all students chose to complete the survey.As a result, we have 211 student survey responses, representing approximately 70% of total potential responses.The course improvement survey included the Attitudinal Learning Inventory (ALI) (Watson et al., 2018) to measure students' perceptions of attitudinal learning regarding the use of DB for CBE (see appendix).This validated instrument examines attitude through four components (cognitive, affective, behavioral, and social).It utilized a 5-point Likert scale (1 = strongly agree, 5 = strongly disagree) (see Table 2).
The survey also contained demographic items, an item asking what course activities were most impactful to the students' attitudinal learning, and open-ended text questions asking what why that activity was most impactful and if their behavior regarding DB had changed since taking the course.The second set of data comes from interviews with individual students to gain a deeper understanding of their experiences.The team decided to use a convenient sampling method as well as the snowballing method to recruit interview participants.One member of the research team reached out to the current competency course instructors and requested them to identify 2-3 students in their courses who might share their experiences.In addition, we asked students to suggest peers to participate.As a result, we recruited 17 students and 14 of them agreed to participate, representing participants ranged from novices to those who had graduated (see Table 3).Three members of the research team conducted the interviews.Each conducted individual interviews with 3-5 participants.All interviews took place online either via WebEx or Zoom.The participants received the semi-structured interview questions (10 total) prior to their interviews so that they could reflect upon their experience in the program ahead of time and revisit their course work if desired.The interviews lasted between 30 to 60 minutes.

Data analysis
The interview data was transcribed independently.The lead researcher selected three interview transcriptions for each researcher to code separately.The research team then convened to share and discuss the themes they identified in the interviews.Through the discussion, four major themes and subcategories of these themes were developed, which established inter-coder reliability (Creswell, 2014).The research team then re-coded the previous three interviews and the rest of the 11 interviews based on the themes and the categories identified.The lead researcher then consolidated the codes.Finally, the research team members crafted a brief narrative for each of the 14 interview participants (see Figure 1).The lead researcher compiled all researcher members' codes.
The research team agreed on four major themes and the subcategories for the themes.
The research team convened and discussed the codes and themes they identified.
The research members wrote brief narratives of the interview participants.

Participant breakdown
As described above, participants included active students who were surveyed at the conclusion of each semester during the study, as well as a smaller group of students and graduates who were interviewed.Table 4 details the characteristics of those who were surveyed.Table 5 gives a breakdown on the interview participants, their ID background, and where they were in the program when interviewed: Interviews resulted in four core themes that described the students' experiences with the DB for CBE.The core themes described their initial attitudes coming into the program, attitudinal learning experiences regarding the DB, perceptions of the DB-earning process, and future plans for utilizing DB.

Existing attitudes
A clear theme amongst the participants was that the vast majority of those interviewed lacked any clear awareness or existing attitude regarding the use of DB for CBE.Of those interviewed, only two really spoke of any significant awareness of how DB could be used for learning, with all others lacking any significant knowledge or perspective towards them or only being aware of them being used in games and on social network sites.Kevin noted that he was aware of their use and already had a positive attitude towards them prior to entering the program, saying: I just heard about it kind of second hand from other IDers who were working on it so that was kind of my exposure to badges, and I think just from that what I saw from the conferences and hearing from other ideas was that it was a great way to like show individual skills….
Sarah had a less positive view, thinking they were best suited for children.It was clear that few of the students entering the program had firmly established knowledge of or attitudes towards DB and for those who had prior understanding, it was not overly specific or strongly positive or negative.What also became clear was that as they learned more about the badges, the concept was attractive to them.

Attractive concept
Another clear theme related to initial attitudes was that upon being exposed to how the program was utilizing DB, the concept was attractive to these potential students and helped to differentiate the program.Four of those interviewed were explicit in bringing up without prompting that the badges were an important reason for why they chose to enroll in this program over others.Kelly described: I had never heard of digital badges, and then I started researching grad school…, and I was looking at different programs… [this program] was the only place I applied and mostly just because of the badges.Because I thought here's a great system where I'm really going to get these specific competencies and… a portfolio that I'm going to showcase when I graduate.
Kristi likewise described how "it was actually one of my deciding factors in going with the program."It was clear that contrary to turning potential students away from the program, the badges helped attract those students who investigated their use in the program.Despite neutral to positive attitudes towards the badges entering the program, the next theme identified the challenges of becoming familiar with the process of earning badges and demonstrating competencies.

Learning the process
A common experience was initial confusion and frustration with learning the process for earning badges.This was particularly noted by participants who were amongst the first to utilize the process as it was refined both after an initial pilot program and again after full adoption.After refinement, additional resources such as instructional videos, rubrics, and worksheets were provided to help students get up to speed on the process more quickly, and this improvement was noted by several of those who had experienced both versions of the badge-earning courses.
It was clear that the initial badge-earning course was confusing for many of the participants which is unsurprising given how conceptually new the approach was, and that as they progressed through the courses, they settled into a comfortable understanding of the work required.Sarah described how the supporting materials helped her to understand the process, saying "I think the best aspect was that the instructions were clear, and I appreciated that there was a model response that I could read."Aanya, also noted how her uncertainty changed over time, relating: "It was intimidating, but once I got started…it was actually pretty simple." The final theme we identified related to pre-existing and entry attitudes, knowledge, and beliefs was the difference in individual personality that some of those interviewed demonstrated in relation to the program.We discuss this next.

Personality differences
Different personalities and approaches were another aspect that became clear through the interviews.For some, the novelty of the badges was attractive and exciting or a good fit with their personality.As John described: When I was shopping for a program…I thought 'that's a neat approach.'I'm sort of the person I think is wired for accumulating certifications.I just like to plan new stuff.So, when I saw that I thought, 'Okay cool!That's a neat way to sort of engage people in a variety of learning.'Kristi also shared an interest and optimism to try something new, saying "I thought it was unique, and I was really excited to work with that badge platform to see what it was all about." On the other side, there were some participants who were more negative in general to their response, noting the additional work was frustrating to them while they were doing it even if they had a positive perspective in the end.Sarah described her love/hate relationship with the badges, saying: It's weird because I think it works really well as an instructional tool, and I would like to implement it, but as a student, I was like 'This is onerous.[laughter] This is driving me crazy.'I mean, so to be perfectly honest with you, when I was doing it, I was like 'this is a humongous pain in the butt.I already did the work now I have to write about the work I did.' Janet had the most negative perspective of the badges, never really embracing the value even though she did perceive it as she planned to utilize her e-portfolio in job interviews.She noted how she was not motivated by the badges but just focused on getting through the program, describing, "I didn't feel like I was earning badges at all…I really thought that I was just fulfilling the requirement because it's a required course-something that I needed to do." While the majority of participants entered with neutral or even positive attitudes towards the badges, there were clear differences in how individuals dealt with the day-to-day efforts required to earn the badges.While final attitudes reflected varying attitudes towards the badges, even those who were more negative did describe positive outcomes they recognized either in products or learning that arose from the badge-earning experience.The next section highlights the attitudinal learning regarding the badges that was evident through the interviews.

Attitudinal learning
The first research question in this study is What are graduate, online ID students' perceptions of attitudinal learning regarding the use of DB for CBE within the program?This question was addressed both through the implementation of the ALI in the survey, as well as open-ended survey questions and through the interviews.
Table 6 shows those responding to the survey indicated that they perceived positive attitudinal learning (1 represented strongly agree with attitudinal learning gains and 5 represented strongly disagree) across all four components of attitude, with the most positive outcome regarding cognitive learning.While the ALI resulted in quantitative results, demonstrating that those surveyed perceived a slightly positive attitude change towards the use of DB, our interviews highlighted richer examples of how those gains manifested.Keeping with the four components of attitude examined by ALI, themes emerged demonstrating attitudinal learning in cognitive, behavioral, affective, and social areas.

Cognitive learning
Many of the participants highlighted the cognitive gains they felt that earning the badges helped provide them, whether that was representative of cognitive growth, an understanding of what competencies IDers needed to master, how the program coursework connected to learning how to be an IDer, or even time to reflect on and recognize learning that was taking place in a fast-paced program.For example, Kelly highlighted how "I have really grown a lot in just my ability to just analyze and self-reflect, and really put into words the things that I'm learning."Kristi likewise noted her growth, saying "…a lot of critical thinking and even problem-solving goes into it to be able to say… I now have this badge to show that I am competent, or I have this particular skill." (connect your ideas) Summer described the value she saw in the evidence embedded in the badges, saying "I appreciate not just being able to demonstrate the competencies but also having artifacts for each of them.The artifacts are meaningful."Jason was struck by the efficacy of the reflection process as well as the peer review in helping to deepen and understand his own thinking after utilizing reflection in his role as a corporate trainer: "…up until actually implementing reflection in my own coursework at [his company], I didn't necessarily appreciate how useful it can belike the actual metacognition in the learning process." Others also highlighted how this metacognitive process and the connection of the badges to professional competencies helped them to understand at a high level what they were learning in the coursework and why.Anja described the value in having a systemic view of her learning and how it connected to her past experiences and goals as an IDer, laying out that: …the [badge] course is actually the means by which learning was made relevant for me for the big picture….The reflections actually pointed at analogous situations for me with my previous work life… solidifying my competency.It was this sort of revelation that basically showed me what current attitudes and desired capabilities an instructional designer would need to have….

Michelle echoed this metacognitive benefit by describing how it helped her to understand what was required of an
IDer by making explicit what she was learning: "I think that they provide a lot of value…because of the way that the competencies are structured.I can figure out…exactly how I'm supposed to use these skills or this competency in my career as an instructional designer." In addition to describing cognitive gains impacting their positive attitudes towards badges and an understanding of the benefits of connecting DB with professional competencies, participants also described their continued use and plans for future use of DB.The survey data also showed that the DB System contributed most towards participants' cognitive learning when compared to the other forms of activities (see Table 6).

Behavioral learning
Program students are often either already practicing IDers or have their eyes on a career change to become IDers.As such, we saw numerous comments indicating that the participants were either already using DB or had the intention to use them as IDers.Anja described that "I'm using it here at work.We don't have anything like this…so I'm using that to formulate a new practice essentially."Kelly, who had no knowledge of DB prior to entering the program, likewise was already applying them in her own practice: "I'm actually incorporating into a course I'm developing right now, that's an Arts Education course for K-12." In addition to those already or intending to use DB as a learning technology in their ID practice, many of the participants were utilizing their earned badges and accompanying reflections as an e-portfolio when applying for ID positions.Despite having a largely negative attitude about her badge experience, Janet, who was not currently working in the ID field, noted her intention to use the badges when interviewing:

I would definitely showcase [them] because I have seen other people saying that… the competency badges helped them to get their jobs. So, I think the concept is really good like having a portfolio so that way you can easily demonstrate to the future employers what you have done so they can see you know the work.
John indicated his intention to keep his portfolio updated in case he pursued a new position, and Cathy had created her own e-portfolio after graduating from the program, using a website and utilizing it and her badges in a successful application and was now an IDer with our University.Emma likewise saw the value, describing how "I interviewed with a company that I was going to try and do a project for, and she asked me to send my portfolio, so I was thinking great I have this portfolio already to go in this badge system." Beyond intentions to use and experience using DB, participants also described evidence of attitudinal learning in how they felt about DB and the process of earning them.Participants perceived that competency and badge resources impacted their behavioral learning more than the other forms of activities (see Table 7).

Affective learning
Numerous participants described a shift from frustration and even anger with the badge-earning process to recognition of the value.Emma described her long journey towards appreciation: …I had this horrible relationship with it, …and I got very angry…initially, it was like it was such a waste of time, and I hate doing this, and it's taking away from other things that I need to be doing.And eventually, I came to see it as something that's really beneficial for me as a student and also as a professional.
For John, he still maintained a certain level of frustration towards the process but gained what he described as "loving the technology that… is such a neat compact format for saying, 'hey I got a baseline skill in this,' and so I thought it was extremely valuable in the workplace."John noted the amount of work required for what was a one-credit hour course each semester as part of his frustration.Sarah likewise highlighted the clear positives that she recognized while also bemoaning the amount of work the peer reviews in particular required and questioning the cost-benefit.
Others highlighted the motivating and empowering nature of earning badges as they progressed towards completing their degree.Kelly noted: I feel like they are very motivating, I like to be able to kind of see like, here's my learning pathway even, and here's the progress I'm making….And so, it really helps me feel like, here's this benchmark that I'm meeting along the way, here's the progress I'm making, here's how far I have to go.It just kind of pushes and motivates me….
She also described how the evidence-based nature of earning competencies helped make her feel more confident, saying: "It's giving me just a lot of confidence I think, in my design work, to kind of know that I can do this.Because when I started this program, I didn't know if I could...." Whether describing enjoyment, increased confidence, or a shift from frustration to appreciation, many of the participants described their experiences in terms that highlighted emotional responses to the DB and process.Related to this was the learning that resulted from their social interaction with their peers and instructors.It is worth mentioning that instructors play a big role in affective learning.The survey data (see Table 6) showed that none of the instructors influenced cognitive, behavioral, and social learning significantly.However, affective learning was significantly influenced by instructors.

Social learning
For many of the participants, the social aspect of the required peer review aspect of the badge-earning process was the most valuable aspect.In the courses, students were first required to have two peers review each piece of evidence being submitted in order to complete badge requirements.After feedback from the initial cohorts, this was later reduced to one.Only after receiving peer feedback and addressing it were students able to submit their work for final evaluation from their instructor.Similar to behavioral learning, participants perceived that the Competency and Badge Resources impacted their social learning more than the other activities (see Table 6).
Emma felt that learning to provide this feedback was very valuable for her, saying "…for me probably the biggest benefit would be learning how to communicate in a positive way….You know…give constructive feedback…."Aanya noted how receiving effective feedback was very beneficial for her, "I really enjoy getting feedback from my peers to improve the reflection piece.It has been just really helpful because it's really made me think about the work that I'm doing and what competencies does [sic] my work demonstrates." Others also highlighted the value of receiving quality feedback from their instructor.Steve found that since his badge evaluation instructors were freed to focus primarily on providing feedback rather than facilitating instruction for the course, he felt more connected to them, saying "The adjunct instructors you have for the badges are usually more engaging and more helpful than the instructors you have for the regular courses.They give more feedback." Emma described how the course feedback she received from the instructors became the turning point for her in recognizing the value of the badge process to her learning, saying "I could see that submissions had gotten better and so I was getting to a point where they were getting approved on the first time instead of you know second and third time…."

Impactful learning activities
Throughout their discussion of attitudinal learning perceived across the four components of attitude, participants identified learning activities within the badge course that were most impactful to their attitudes toward the badges.Our second research question focuses on this: Is there a difference between different preferences in learning activities with respect to perceived attitudinal learning gains?
Table 7 shows that those who reported peer review, self-reflection, or instructor feedback as the most impactful learning activity in their badge course perceived more attitudinal learning gains than those who didn't.However, resources about DB and the system did not affect learners' perceptions of attitudinal learning gains.Among the activities, it was found that Peer Review was the single most impactful activity by 39 participants with several others picking Peer Review along with other activities as impactful for their learning experience.In the survey, they mentioned that Peer Reviews provided "additional insights that were not previously considered" and that they provided an "alternate viewpoint," and "valuable feedback" besides enriching the learning from "providing feedback" to others.One participant said, "The resources were often obtuse and not clear, especially to a person like me with little knowledge of ID before the program.The peer reviews allowed me to compare myself to others to see how others were meeting the competencies.Reflecting on how I met them allowed me to better understand the competencies by working them out somewhat on my own." While the participants described a variety of positive attitudinal learning gains they recognized through their experience, they also discussed other experiences with the badge-earning process, including some aspects that were less positive.

Process experiences
It was clear that many of the participants recognized the benefits to their learning that the badge process facilitated; however, it was also common that they had certain frustrations with the process, had particular elements within the process that they saw as strengths in moving them towards a positive attitude, and they had specific suggestions for improving the process of earning DB for CBE.

Process frustrations
While most participants expressed a very positive view of the badges, frustrations with the process of earning the badges were common, even amongst those who held a strongly positive position on the badges overall.Frustration was more typical amongst those who were from the early cohorts and had completed their degrees.For example, Sarah, who had graduated, described her frustration with the amount of time and effort required for the peer reviews, saying, "I think the biggest annoyance was all the peer reviews."Likewise, John, who had also finished, described some difficulty, or what he called "hiccups in the process" with some of his instructors and having to take time during a busy degree program to improve work from the courses that he was using to earn his badges: "…there were a couple of [badge course] instructors who wanted to relitigate….It was a force of frictions sometimes to go through this sort of recertification." Early feedback from some of the cohorts had already resulted in changes to the process, such as improved instructional materials regarding the badge-earning process, a reduction of required peer reviews from two to one, and new training materials for badge course instructors, including evaluation rubrics to improve consistency of evaluation across instructors.
While we did observe expressions of frustration from the younger cohorts with the additional burden required to earn badges alongside the already hectic coursework, outside of Janet who, while acknowledging that she believed she would benefit in her job search process by utilizing the badges, expressed a rather negative view of the badges, the rest of those in the more updated cohort were less expressive with their frustration and did not identify these two aspects so specifically.Jason and Michelle did note the significant work, for example, but stressed their overall positive views.

Process strengths
The participants did identify different strengths of the process with their experience in the badge process.For some, it was the social learning that occurred through the feedback they received from their peers, and the sense of satisfaction of seeing their growth over time as their early struggles to get artifacts approved faded, and they were able to recognize the growing quality of their work and ability to align components with competency requirements.Others highlighted the process of communication with instructors while seeking approval.For Michelle, the clear structure of the process was very important to her: I think that the class structure is the biggest benefit…the due dates are holding me accountable, focusing on one at a time helps me accomplish them… because it's structured this way…it makes me able to do them.And I think I might have less of a positive attitude towards them if I had to manage all of it on my own.Aanya valued the personalization that came with the ability to not just only choose what coursework would be utilized to meet badge requirements but to also have the option to incorporate professional work that was external to the program and university.She described that "…the fact that we can choose from our professional work or academic work makes it even more impactful because it makes me think about the work that I've already done."While participants pointed to a variety of aspects of the program that they felt were most helpful, they also had ideas for how to improve upon that process.

Process improvements
Participants had significantly varied suggestions for improvements to the badge-earning process.Several highlighted aspects of the badge platform and how it was utilized.Sarah created her own spreadsheet for tracking but would like the platform to include "an online trackable sheet that allows me to easily see which badges I had completed and which I had not all in one screen."A few of the participants had some confusion with the four supra-badges, which were used as a hierarchical organization of the sub-badges, and Cathy would have liked better color coordination amongst the hierarchy to help her better visualize the relationships.Jason had some visual problems and found the lack of accessibility of the platform challenging.
Others highlighted specific changes in how the program was conducted.For example, Kevin would have liked reflections that were not required to be written but could incorporate a video recording of a presentation.Kristi found it problematic to still work on earning badges while completing her required practicum at the end of the program, which involved professional work with a client and would have preferred different timing so that she could be more focused on the practicum.Finally, the participants also had differing thoughts on how well badges were being accepted by employers and how well they fit in their current context.

Questions of fit and employer buy-in
While the participants were largely very positive about DB and how it impacted their learning, some were less positive about utilizing badges themselves as IDers or even leveraging their earned badges for job seeking.For several, the badges were not a good fit for their current context.For Sarah, working in an institute of higher education, DB had been discussed in a joking way and condemned as a juvenile approach to learning.She described the faculty she worked with laughing at the thought and noted that "…my department has sort of dismissed them out of hand…." Others also saw their contexts as not being a great fit.Emma described how the size of her company would make it difficult for her to individually be able to apply them: "I work in a bigger company.and we are in multiple states.It would really be hard, I think, for one person to keep track and oversee a badge system… it would require kind of the whole system to change."Other participants also described the badges as so cutting edge, that they felt that it was unlikely that they would be able to gain much traction in their current workplace.
This also was an issue for some of the participants in using the badges they had earned as a portfolio when seeking a position.For Kevin, he did not get the sense that it was something employers were aware of or looking for, saying: …it just seems like getting buy-in is tough from employers, and in general, I feel like everything is still focused on the degree… I don't see a lot of employers looking for badges….I definitely think that they're kind of asking for it, they just don't know it yet.
This mindset was echoed by Steve who felt employers would assume he had the competencies already if he had earned the degree.For him, and several others, they saw the most value in the optional technical badges offered by the program.Separate from the required competency badges, technology badges embedded instructional projects to teach learning technology platforms.Numerous participants felt these badges were more desired by employers than the competency badges.

Discussion
A clear indication from the data was that participants expressed positive attitudinal learning regarding the use of DB for CBE across all four attitudinal components.The majority of students lacked significant knowledge or an existing attitude regarding DB prior to finding the program, and for some students, the inclusion of DB was a significant factor in their decision to join this program over others.
The positive attitude indicators highlight participants grew to recognize how the badges impacted their learning, they were able to make better systemic connections between their coursework and their professional goals, and they were able to metacognitively reflect on their understanding and critically interpret their approach to solving ID problems, an important aspect of developing ID expertise (Ertmer & Stepich, 2005).
However, for a few learners, the badge-earning process was repetitive and meaningless.Individual personality may have played a role as there were indications that some students were focused on the degree rather than learning.Even amongst those with positive attitudes, there were indications of some frustrations with the process.
A clear implication, however, is the need for continued refinement and incorporating student feedback to promote efficiency and balance learning with effort.Further investigation is needed into how to better communicate the inherent value of the approach and achieve more buy-in from students with this viewpoint, a result echoed in the literature in Stefaniak and Carey's (2019) recent multi-case study.Additional research tracking entry attitudes and potentially examining the relationship between personality and attitude could prove helpful as well as experimentation around messaging with the badges in both recruitment materials and resources.
With affective learning, numerous students expressed the motivational impact of the badge process, an outcome present in existing literature (Ahn et al., 2014;Anderson et al., 2013;Denny, 2013;Glover, 2013)-how peer feedback and approval by their instructors helped them to gain confidence (Newby & Cheng, 2020) and feel prepared for professional work.Participants described how earning badges along the way helped them better track their progression and understand their learning pathway (Higashi et al., 2012).Some identified with earning a visual representation, and for others, the transparency of the connection between coursework and professional competencies was key in highlighting relevance to professional goals.
Given some of the frustration with the amount of work required, and the time spent on earning badges; however, it is clear that achieving an appropriate balance is key-as is designing a process that helps students to understand and be influenced by the motivational elements early on in the process to help that motivation pull them through what is a significant amount of effort.The repetitive process of revisiting past coursework was identified numerous times as demotivating.An implication could be targeting service-learning opportunities in the coursework, particularly for those students who are not yet professionals in the field so that students could feel that improving prior work is not just for their learning but also for real-world benefit to actual clients.Students in the program do have a required practicum, but it is only a single course.
The social learning aspects of the process were highlighted in the survey as the most impactful element to the participants.Continued improvement of the process of training and evaluating students on the quality of their feedback as well as continuing to improve the consistency of instructor feedback and evaluation approaches across course sections could also further improve what was already highlighted as a strength of the process.
With behavioral learning, it was interesting that while there was consistent evidence of behavioral learning, the intent to adopt DB in their own ID practice was limited.There were significantly more discussions of utilizing the badge portfolio as part of job search strategies, something readily found in the literature (Herman & Kirkup, 2008;Kilroy, 2017;Lin, 2008;Wakimoto et al., 2019;Wright et al., 2002), and numerous participants discussed keeping an updated portfolio.It seems clear that while students were attracted to the program through its novel use of DB for CBE and recognized the many benefits it provided, they believed that in this case, academia was ahead of much of the rest of society in its use of badges and therefore this limited their belief in fully adopting it themselves.
Ultimately, significant implications are that the students have improved learning through this process and most recognize the added value.However, the details of the process are key to both the effectiveness and their attitudes towards the program and the process.Every aspect of that process that promotes frustration, including technical limitations of the badge platform itself, and the perceived usefulness of badges as a credential, impact the students' willingness to put forth effort in the process; the effort can also impact the effectiveness of their peers' learning given varied quality in peer feedback.
Additional efforts in reaching out to corporations hiring graduates to get buy-in are key to not only improving student acceptance of the badges as useful beyond learning but professionally as well as improving the likelihood that they will utilize the technology themselves as IDers as most feel powerless to initiate such a demanding and complicated process in isolation without full support from management.How to influence the attitude and mindset of human resources and ID departments in corporations around the utility and efficacy of the technology itself is an area in need of significant further exploration.

Conclusions
There are several limitations to this study.Study results are not generalizable due to the restricted nature of the case study method.The short time length of one year for the study is also a limitation, as it would be helpful to follow a few cohorts throughout the entirety of the program to see the students' overall experience within the program; although, we did mitigate this by also interviewing those who had completed the program.The ALI survey relies on student selfreported perceptions of attitudinal learning rather than directly assessing attitudes.Furthermore, while we interviewed 2-3 students from each level of badge courses, more insight might have been gained if a larger number of participants were interviewed.
While there has been more badge literature published recently, related to such areas as goal setting, motivation, peer review assessment and teacher professional development, as a relatively novel technology, future research would support improved implementation and guide adoption.CBE systems in particular need further study.As active engagement and systemic collaboration of all stakeholders within higher education is the only way to facilitate a sustainable and effective continuation of a badge system, it is important to explore how we can improve stakeholder mindsets and buy-in, particularly with employers.
Future efforts will need to also include an emergence of standards within higher education on how the badges from the various institutions can be assessed and validated to improve the acceptance of badges as legitimate credentials.The long-term use and impact of badge curricula and programs in higher education also need to be further explored by researchers in order to identify the most effective and impactful utilization of the technology.This study provides an examination of one program and how students perceived their experience with that program to better evaluate how well the process has met program goals and inform others of one approach to leveraging DB for CBE.

Table 1 .
Competency course sequence

Table 4 .
Survey participant characteristics

Table 5 .
Interview participant backgrounds

Table 6 .
ALI results

Table 7 .
Results of independent sample t-tests to determine whether perceived attitudinal learning gains differ according to the perception of the most