Top 10 Research Findings 2012 # 3

Once again we’re counting down!

 

#3 Investigating Real Time Predictors of Engagement

 

When we design games, one of the most critical aspects of the design is the procession of the game.  When we are talking about learning, we tend to call it scaffolding, or increasing levels of difficulty.  In games we often use levels to depict the gradual increases in skill required to proceed within a game.  This study like Peter’s is an outstanding dissertation in the area of games.  This study also had several studies, and the results all culminated in some interesting finds about engagement in games.  The area that I was most interested in however, was the work that Sharek did in order to get to his main research points.

 

 

 

The study employed a game named Gridblocker, a tile based puzzle game in which a player must move blocks around until those blocks are placed on a goal space.  The game has many levels that represent puzzles of increasing complexity.  Sharek was interested in engagement, personal affect, cognitive load, and the amount of time a player engaged with a game.  In order to accomplish this he built 3 conditions into his study.  In the first condition players progressed through the game in a linear fashion, moving onto a higher level when the previous easier level was completed.  In the second condition the player had the choice of whether to move onto a harder level, or to play a level that was easier than or equally as challenging as the previous level completed.  The final condition leveraged an algorithm developed by Sharek which allowed the game to adapt to the player’s skill set, determining the difficulty of the upcoming level for them.

 

The participants in this study were recruited using Amazon, and included 340 adults.  While a preliminary analysis did not report significantly higher engagement, personal affect or decreased cognitive load for those in the choice based or adaptive conditions, the researcher performed a secondary analysis applying the adaptive algorithm to the choice and linear conditions.  The results were fascinating!  The secondary analysis indicated that players in the choice condition chose to play easier levels than they were capable of completing.  Additionally, players in the adaptive condition were able to achieve greater levels of difficulty than either the choice or the linear conditions in the same amount of time.

 

What does this mean to us?  Well, it means that our learners might not be challenging themselves enough when they are allowed to choose the level of difficulty.  It also implies that using an adaptive leveling strategy might allow players to get to harder levels faster than a linear model.  All good things to consider when trying to determine your own scaffolding/leveling strategy!

 

Sharek, D. J. (2012).  Investigating Real-Time Predictors of Engagement:  Implications for Adaptive Video Games and Online Training.  A Dissertation submitted to North Carolina State University.

Top 10 Research Findings 2012 #5 & 4

This is a continuation of several earlier posts, broken up to keep you awake while you’re reading!

#5 Influences of Game Quests on Pupils in Math Learning

When we think about quests, its easy to get all super geek and reference Dungeons and Dragons and those sorts of role playing games.  But quests are an important part of many games that we play today.  Video games like Call of Duty often call them missions, but the result is the same.  A quest uses a mini-story line, often inside a larger story line to give a player a sense of purpose and goals.  This study investigated the use of quests on math learning.

While I think the results of this study are quite interesting, I have to admit that the research was not the first thing to draw me to it.  It was the population being used.  We often hear about games initiatives in Science, Technology, Engineering and Math (STEM) geared towards US students who have fallen behind their international counterparts.  This study was designed and executed by a team of Taiwanese researchers, making it a fascinating find.  Taiwan?  Aren’t they great at math in Taiwan?  Yes.  Yes they are.  So why research a math game for students who are already great at math?  Because even though the Taiwanese students who participated in this study were high math performers, they reported low positive attitudes towards math learning.  Now isn’t that interesting?  That the reason for high performance in math may not be a motivational construct?  I thought so.

The game used in this study was designed specifically for this research.  It was titled “My-Pet-My-Quest” and was modeled after games that involve caring for adorable fuzzy creatures.  The researchers implemented a quest delivery mechanism to provide three elements: learning objectives, learning tasks, and rewards.  Each quest was related to goal orientation, and gave the students specific quests to accomplish.  The participants in this study included 53 4th grade students.  The researchers were interested in the influences of game quests on math learning in terms of the students’ perception of enjoyment and goal-pursuing.

The results of this study indicated that the quests did have a positive impact on the students.  Quests positively impacted the student’s perceptions of their experience including enjoyment, goal orientation and goal intensity.  The students who played the game with the quests also demonstrated more active participation and reported more enjoyable experiences.  The general conclusion was that quests whose goals align with learning objectives were favored by students.

When designing games for learning, its often very easy to think of a single layer game.  To create a story line and a linear experience.  This study shows that positive results can be elicited by using a more complex design that allows for quests and goals associated with those quests.  A great finding for anyone interested in implementing games!

Chen, Z.; Liao, C. C. Y.; Cheng, H. N. H.; Yeh, C. Y. C.; & Chan, T. (2012).  Influences of Game Quests on Pupils’ Enjoyment and Goal-Pursuing in Math Learning.  Educational Technology & Society, 15(2), 317-327.

#4 Cooperative Vs Competitive Goal Structures in Learning Games

If I were to ask you what makes a game a game, I would expect you to answer with a few archetypal game characteristics.  Score, challenge, competition, story, characters.  Those are typical responses.  While my list may differ in some ways, I’m always interested in how each characteristic might contribute uniquely to learning.  This study does just that!  This study (admittedly) is of particular interest to me because it was conducted as part of the dissertation requirement at my Alma matter by my best friend Peter Smith.  This game is additionally near and dear to me because it uses one of the games that Peter built for me in my role at DAU.  That game, Acquisition Proposition was designed to be a part of our casual games initiative, which allowed games to be developed that were based on the core learning objectives of the acquisition process.  This game models a typical resource management game in the style of a Diner Dash variety of game.

This study taps into some commonly held beliefs about cooperation and competition in games, and does much to inform us of how best to leverage games for learning!  Peter (or Dr. Smith as he requires that I call him) took Acquisition Proposition and modified it to be cooperative in which two players might work together or to be competitive in which two players competed against each other for a higher score.  (please note, he modified the infrastructure of the game to delivery it differently, he did not change the game or its learning objectives).  Peter recruited 160 undergraduate students and ran 3 experiments using the cooperative game, the competitive game, and a text based document of the content of the game.  Peter used pairs of students to play the game side by side.  His hypothesized focused on learning, motivation types, and player self efficacy.

Study 1 tested the results between the cooperative game pairs and the competitive game pairs, but provided the students with no particular instructions.

Study 2 tested the results between the cooperative game pairs and the competitive game pairs, but asked the players assigned to the competitive group to try to get a higher score than their game partner.

Study 3 tested the results between the cooperative game pairs and the competitive game pairs, but informed students within the competitive group that the winner of that pair would receive a $10 gift card.

The results indicated that all players in all groups had significant learning increases (yay for Acquisition Proposition).  In Study 1 players within the cooperative game group reported a larger increase in intrinsic motivation.   Studies 2 & 3 elicited increases in intrinsic motivation for all groups, a surprise as it was expected that the intervention (gift card or told to compete) might elicit an extrinsic motivation in its players.  Additionally, winners of all competitive games reported higher self efficacy (their own feelings about their ability to learn) than losers.  Finally, winners of the game who received the gift card reported higher self efficacy than those in the cooperative games group within Study 3.

Deep stuff right?  Lets break it down.  Traditional cognitive theory would suggest that cooperative players would be intrinsically motivated (motivated to learn in order to be successful) and that players who were put into a competitive situation would be extrinsically motivated (motivated to win).  Additionally, it would be normal to expect losers of competitive games to have lower self efficacy.  While some of these results were found, their absence is just as informative.  I’ll write more about this topic later, but in the meantime, if you’re going to design a game, make sure you read this study and draw your own conclusions about its implication!!

Smith, P. A. (2012).  Cooperative Vs Competitive Goal Structures in Learning Games.  A Dissertation submitted to the University of Central Florida.

Top 10 Research Findings 2012 #7 & 6

I recommend you start at the beginning, but if you’re like me and you have no real use for directions, proceed!

#7 A Study on Exploiting Commercial Digital Games into School Context.

When many of us think about implementing games into our educational and training contexts, its really easy to wonder if there is a game out there that meets our needs.  While it can be tempting, I find that more often than not a game that was designed to teach something else will, indeed, teach something else.  There are, however, opportunities to use games for higher order thinking skills and soft skills as those are generalizable.  There is also the potential to modify or mod a game to suit one’s needs.

This study used a version of a commercial game that they felt was most similarly aligned with their learning objectives.  While I can’t attest to their use of any game, the game selected for this research was a very popular game, The Sims.  Specifically, this study used a version titled, The Sims 2: Open for Business.  In this study, the researchers hypothesized that the use of this game could impact achievement in math, general education and that the game could affect the attitudes of its players towards math teaching and learning.

The study included 56 participants who were aged 13-14 and lived in Athens, Greece.  The randomly assigned groups were pre-tested to ensure that their math scores did not significantly differ prior to the intervention.  While results did not demonstrate higher or lower achievement in math, there was a difference indicating an increase in general education for the group that used the game.  Unfortunately, there was no difference in attitudes towards math teaching or learning.

While these results seem lackluster, they do affirm the difficulty in using a commercial game for a purpose it wasn’t specifically designed for.  This may seem a bit contrary, considering the previous results indicated for the commercially available Peacemaker game, but its important to remember that Peacemaker was designed to have the impact the researchers tested, while the Sims 2 was not.

Panoutsoploulos, H. & Sampson, D. G. (2012).  A Study on Exploiting Commercial Digital Games into School Context.  Educational Technology & Society, 15(1), 15-27.

#6 The Effect of Action Video Game Experience on Task Switching

The ability to switch tasks quickly and effectively has grown more important as technology and workplaces become more topically diverse.  Certainly today’s military is concerned with this ability, but many types of organizations can benefit from this skill whether it be transitioning from one meeting to the next or the ability to respond to emergencies during high workload times.  This study expands on a line of research dedicated to the impact of video games on cognitive/perceptual abilities.

This particular study focused on task switching, and its researchers sought to better understand the potential impact that action based video games might have on it.  The researchers conducted a series of 4 studies:

Study 1 focused on whether experienced action game players demonstrated an advantage in a modality switch (motor vs voice response) in a response time task.  The results showed that action game players were better able to switch modality.

Study 2 focused on replicating the findings in Study 1 with a more cognitive task, which resulted again in action game players demonstrating greater agility.

Study 3 increased the difficulty of the cognitive task in Study 2, and again resulted in action game players demonstrating an advantage.

Study 4 sought to determine if action games could be used to remediate the non gamers to reduce their response times.  In this study the non gamers were asked to play approximately 50 hours of either an action game or a simple game.  While both groups improved, the group that played action games demonstrated increased improvement.

These results of this study indicate that not only do game players demonstrate higher ability to switch tasks, but that video games can be used to train ability to switch tasks!  This is a really important finding as it demonstrates the ability for those using games to increase cognitive and perceptual abilities!

Green, C. S.; Sugarman, M. A.; Medford, K.; Klobusicky, E.; & Bavelier, D. (2012).  The Effect of Action Video Game Experience on Task-Switching.   Computers in Human Behavior, (28) 984-994.

Top 10 Research Findings 2012 #9 & 8

This is the to be continued of the Top 10 Research Findings in games posts!  If you’ve landed here without reading that post, I recommend you go back and take a peek at the evolution of the list and #10!

#9 The Relationship Between Video Game Use and a Performance Based Measure of Persistence.

Thomas Edison once said, “I haven’t failed.  I’ve found 10,000 ways that don’t work.”  I think that I have a lot in common with Mr. Edison in that respect, but, if nothing else I am persistent.  Not to say I’ve accomplished all that much, but I am not one to give up.  When we look at the ideal candidates for our jobs, we often think about what they already bring to the table.  What they have demonstrated they are capable of.  But, maybe we should be looking at how determined they are to learn what we are presenting them with.  Anyone who tackles complex problems fails.  Its inevitable.  But the desire to keep trying, the persistence of that individual is often the means to their success.

What does this have to do with games?  Turns out, quite a bit.  If we knew that playing games could increase our willingness to try and try again.  Our grittiness (as some researchers have called it).  Our persistence, this would be valuable.  Well, this piece of research has begun to investigate just this!

In their research, Ventura, Shute & Zhao investigated whether experience with video games was related to an increased ability to persist on a difficult task.  The authors additionally developed behavioral tests designed to improve upon self reported persistence.  When put to a test for validation of their behavior tests, they did find a small but significant relationship between the time committed towards solving a problem and a participants self reported persistence.  While this requires further validation testing, it does open the door to creating new methods for evaluating this interesting variable.

The findings of this research also found that participants who reported being game players dedicated more time to difficult problems.  This indicates that experience with video games did have a relationship with persistence.

Sounds minor right?  Well, its not.  This is the kind of research that I love to report on, its investigating the tip of the iceberg of what could be a major contributing factor to complex skill development!  Of course, further research is necessary to determine if this effect is transferable to other types of complex problems and to parse types of video games that these participants were referencing in their reports of play.  But still, great news!

Ventura, M., Shute, V., & Zhao, W. (2012).  The Relationship Between Video Game use and a Performance Based Measure of Persistence.  Computers in Education, (60)52-58.

#8 Learning to Stand in the Other’s Shoes.

What did your mom always tell you not to talk about at the dinner table?  Religion and politics right?  Well, chances are that your mom said that because these are topics that people identify with so deeply that virtually no level of ration or discussion will change their viewpoints.  But….what if you could provide someone with an experience that allowed them to see things from the other side?  What if you could allow them to make decisions and see the consequences of those decisions in a contextually relevant way?

This is exactly what the researchers in this study accomplished by leveraging the game Peacemaker.  Developed by Impact Games, Peacemaker is a game that allows you to play a role on either side of the Israeli-Palestinian conflict. Within this game, you play either the Palestinian President or the Israeli Prime Minister.  There are varying levels of difficulty, and your goal is to create peace.

In this study Gonzalez, Saner & Eisenberg hypothesized that using the game would reduce the effects of religious views and political affiliations on decisions to reduce conflict.  They enlisted 42 undergraduates within a course on political topics.  The game was used twice within a semester with each participant playing each role twice.  The role they played first was randomly selected, as was the level of difficulty.

The results indicated that while peace was the goal of the game, most students did not achieve it.  They also found that the role a participant played did have an impact on the amount of time spent with the game, with the role of the Palestinian President yielding the higher time.  Most interestingly however, the Religious affiliation of the student was correlated with the attainment of peace in the first administration, but not in the second.  The same result was found for Political affiliation.  While this might have been additionally mitigated by the amount of time between the two administrations or even the content of the course; the results are powerful.  The use of an experiential game in this study was enough to impact the religious and political views of its players. That’s a big deal.

Gonzalez, C.; Saner, L. D.; & Eisenberg, L. Z. (2012).  Learning to Stand in the Other’s Shoes:  A computer video game experience of the Israeli-Palenstinian conflict.  Social Science Computer View, Sage Publications.

Top 10 Research Findings 2012 #10

A few years ago, it was particularly challenging to find real empirical research related to the use of games in education and training.  Back then my PhD advisor Clint Bowers and I started an annual presentation of the Top 10 Research Findings in games in order to help raise the visibility of the few results that existed.  Over the past 6 years, the amount of good research being conducted has grown exponentially.  What started out as a hunt for 10 articles has slowly become a huge effort, with many great studies no longer making the cut.  Clint and I typically gave this presentation at the DoD Gametech conference, which allowed us to focus on the studies that were most relevant to military training and education, but this past year I was invited to present at Tech Knowledge, bringing a new element to the down select.

In preparation for this presentation, I first pull any research that is related in any way to the use of games, then I select only the papers that include empirical findings.  I then read each article and determine which studies resulted in the most relevant results for the audience I am addressing.  The list here includes a hybrid of both the Gametech and Tech Knowledge presentations, as they both had results that are applicable to anyone interested in the use of games.  They  are in no particular order, and were all published in 2012.

As you read the synopses provided here, I advise you to remember that we are but two games researchers who are trying to find the most salient results.  I have not always included all of the findings of a study, but instead extracted the results most pertinent for discussion in our industry.  I also highly recommend that you take the time to read the articles and reach your own conclusions about their results.  Everything is subject to interpretation, and here you will always get mine!

Drum roll please…..

#10  The effects of learning style and gender consciousness on novice’s learning from playing educational games.

Now, anyone who knows me knows that I have a few hot-button phrases that I despise.  Learning Styles tops that list.  I don’t believe in them.  Research has indicated that they don’t exist.  I could write an entire post on them (and perhaps I shall).  When I found this study the title piqued my interest immediately, and I assumed I would have a good chuckle then move on.  I was surprised to find that this study was not only done very well, but with exceptional vigor.  Also, I was very happy to learn that the learning styles represented in this study were Kolb’s learning style approach, with a specific emphasis on two groups, not the learning styles that I stand so strongly against (you are not a visual learner, nobody is).

In this study, 122 eighth graders used a custom flash development tool to learn programming through game play.  They were tested for two types of learning style in particular – Divergers who were best at viewing concrete situations with multiple viewpoints and Convergers who were best at finding practical uses for ideas.  The participants were also surveyed on their gender consciousness to determine the gender roles, traits and feelings of gender equality or inequality that they possessed.  The study was designed to determine if their style and gender consciousness had any impact on their ability to comprehend programming concepts, their project performance, and their motivation.  The results were complicated and mixed.  First, both intrinsic and extrinsic motivation was similar and positive for all four groups.  Second, it was found that convergers comprehended abstract programming principles better than divergers, but comprehension was not impacted by gender consciousness.  Next, it was found that divergers with low gender consciousness outperformed high gender conscious divergers on project performance.  Convergers of both gender consciousnesses groups performed equally.    Finally, high gender conscious convergers outperformed high gender conscious divergers on project performance, but there was no difference between divergers and convergers of low gender consciousness.

That’s deep right?  So what does it all mean?  Kolb’s Learning Styles did indeed impact programming comprehension.  Gender consciousness impacted divergers and convergers differently.  This is most relevant for teachers and trainers who are interested in using game development to teach complex processes like programming.  Those teachers and trainers should be aware of the learning style and gender consciousness focus of their students and try to mitigate the impacts of their traits on the learning!

Wang, L. & Chen, M. (2012).  The effects of learning style and gender consciousness on novice’s learning from playing educational games.  Knowledge Management & E-Learning: An International Journal, 4(1), 63-77.

I’ll keep posting these, but didn’t want this post to get too long.  Stay tuned for #9!

Games Movements

In this post, I’m hoping to clear up some of the phraseology within games.  While I will save “gamification” for another day, I wanted to make sure that you understand the “movements” that surround games.  There are several types of games movements that games that are not used solely for entertainment purposes may fall into.  Each of these movements has clearly defined goals and their own communities, and some games might cross over from one movement to another.

seriousheading

The most obviously relevant movement for our purposes is the Serious Games Initiative.  This initiative initially started as games designed for ‘serious’ purposes, but has since evolved into games for education.  Serious games has become an acceptable term for educational games as it hosted conferences dedicated to the use of games in education and training for many years.  The Serious Games Initiative, founded by Ben Sawyer has remained a community for designers, developers, academics and practitioners who are interested in the use of all types of games and game technologies in their learning endeavors.  Serious games span all age groups and are related to all types of educational goals.  The most interest recently has been related to Science, Technology, Engineering and Math (STEM) games.  Many federal and private organizations have released requests for proposals for games centered on the topics of STEM in an attempt to motivate students to pursue STEM related careers.  The recent interest in STEM games is attributable to the findings of the 2010 report to President Obama which found that students in the United States continued to fall behind in STEM related career intentions, setting this country into a position that will prevent it from continuing its technological and scientific prominence in the world. 

Other serious games topics might range from K-12 to adult learning games running the gamut from early literacy skills to assembly line fast food training.  Serious games have been used in the military for decades, but have also continued to serve as interesting forms of media that are ingrained in our culture.  One of my favorite games currently is Dragon Box +, a game that starts out with evolving monsters by matching tiles from one side of the screen to the other, but through multiple levels begins to reveal itself as a game to help students understand balancing an algebraic formula.  It is aesthetically pleasing, easy to play, compelling, and accomplishes the rare task of actually teaching. 

 

GamesChange0.100.100.0

 Another movement that has gained momentum in the games industry is Games for Change.  Their self-assigned mission is to “Catalyze Social Impact Through Digital Games”, which means their games focus on gaining a new perspective of society and the issues it faces.  While some of the games that remain a hallmark within the Games for Change movement involve experiencing and learning about different perspectives like Peacemaker by Impact Games, a game that allows a player to make critical decisions from either side in the Iranian/Palestinian Conflict; some games actually allowed their players to make positive impacts on the world.  Foodforce, developed in 2005 was developed by Konami in partnership with the United Nations World Food Programme and actually created donations to world food aid in impoverished areas.  Other types of games might focus on community awareness, safety, food handling, farming, the environment, elections, the national budget, saving sea lions and just about any social issue you might think of.  The Games for Change movement is currently co-led by Asi Burak & Michelle Byrd who host an annual festival in New York City in which hundreds of attendees from various types of commercial and federal organizations can come together.

 games-for-health

 The last major movement in games is the Games for Health Project, which was started as a subsection of the Serious Games Initiative, but will host its 9th annual Games for Health conference in 2013.  The types of games that are targeted in this initiative have some crossover with Games for Change and Serious Games, but are all aligned with health.  Key examples of this initiative might include Re-mission, a game about designed by HopeLab to increase understanding of the disease, its treatment, and to refocus kids and young adults with cancer on the psychological aspects of fighting the disease.  In 2010, First Lady Michelle Obama announced the first Apps for Healthy Kids contest, urging game and app developers to design apps geared towards helping the fight against childhood obesity.  The game winner of the first contest was Trainer, a game developed by students from the University of Southern California which allowed players to train with their creature pets.  Other games within the Games for Health Project focus on delivery of health care and medical training while others took advantage of advanced technologies like the sensor based game Zombies, Run!, designed by Naomi Alderman which tracks a players runs and helps increase speed and distance by incorporating zombies into their running routine. 

All of the movements for games have seen the development of amazing games that demonstrate innovation, creativity, and a true desire to serve as a benefit to learners and individuals.  The types of games used in these movements vary widely and can encompass a large variety of play dynamics and styles.  Regularly understanding the state of the industry in all of the movements can help organizations stay informed on what advancements and types of innovative uses are being accomplished, so try to keep an eye out!

Getting Ready for Games

img_gameon_brand

 

The single most important thing that any person can do to better understand games could never be captured in a blog.  While this post was written to prepare its readers to have the highest probability of success in developing and deploying games in to their educational and training programs, it cannot replace actual experience.  If I could make a game about making serious games, I would, but the potential of games would not be best served in helping you understand the minutia that go into the design and development process, or even the determination of if a game could help.  Therefore, you must play.  There is no substitute for understanding how games work than to actually play as many types of games as possible.  This is not or should not be a painful assignment.  You can call it “research” for your upcoming project. Everybody plays games in one form or another, but few people tend to classify themselves as gamers.  We are all gamers.  I don’t know anyone who has never played hide and seek, or go fish.

There are literally hundreds of thousands of games out there, just waiting to be played.  Play some board games, play some card games, play some online games, play some console games.  You don’t have to own a console, just walk into a store that sells them and they are likely to have some demo games out on their showroom floor that you can play.  As you play each game, think about what the game was designed to do.  Think about what you learned from the game.  Ask yourself what you could transfer from that game into your life.  Get involved in the games community.  Share your stories. Tell others what you are doing.  Ask them what their favorite games are, and when they tell you they don’t play games, remind them that there are many types of games that don’t need to be plugged in.  There is no better preparation than to dig right in and try something new.  After all, if you want people to use your game, you better be prepared to play it. 

A Game of Phones has won the “Best Business Process”

November 2011

Developers Hybrid Learning Systems & Czarina Games are pleased to announce A Game of Phones has won the “Best Business Process” award at the eLearning Guild’s annual conference, DevLearn 2011.  A Game of Phones competed with over 50 submissions at the DemoFest event hosted at the conference in Las Vegas, Nevada.  Hundreds of votes were cast to determine the winners in several categories, and notably A Game of Phones was the only non-digital submission to ever be recognized by the eLearning Guild.   www.agameofphones.com

precio cialis

viagra 50mg

viagra online

Acquisition Challenge has been named a finalist…

October 2011

Dr. Sanchez is pleased to announce that Acquisition Challenge has been named a finalist at the Interservice/Industry Training Simulation and Education Conference’s Serious Games Showcase and Challenge competition in Orlando, Florida.  Acquisition Challenge was commissioned by Dr. Sanchez in her role as Games Czar for Defense Acquisition University and was produced by the Co-ADL Lab.  This marks the third consecutive year that DAU has made the finals in the highly competitive challenge, with previous submissions Procurement Fraud Indicators and the IPC CPI game making the finals in 2009 & 2010 respectively.  https://clc.dau.mil/games

Acquisition Proposition

Acquisition Proposition is a mini game designed to help Acquisition professionals better understand the key component of the Acquisition Process.  Acquisition Proposition is styled after Diner Dash, a resource management style of game.  Players move through increasingly difficult levels in which they are required to move Defense Acquisition Programs through the milestones of the Acqusition Process.