class: left, bottom, title-slide # Module 2: Systems Analysis ## EME6357: Evaluation of Instruction & Training in HPT ### Dr. Bret Staudt Willet ### September 22, 2021 --- class: inverse, center, middle **View the slides:** [bretsw.github.io/eme6357-module2](https://bretsw.github.io/eme6357-module2) --- class: inverse, center, middle # Evaluation Plan feedback --- # Evaluation Plan, Part 1 <img src="img/group-project.jpg" width="560px" style="display: block; margin: auto;" /> -- - Overall, fantastic work -- - Work on developing your arguments (details, hyperlinks) -- - Topic sentences: start with the point, then add evidence --- # Evaluation Plan, Part 2 <img src="img/group-project.jpg" width="560px" style="display: block; margin: auto;" /> -- ### Systemic Analysis of a Training Program -- - Create a systemic logic model for evaluating your training program -- - Look at past examples --- class: inverse, center, middle # Flipgrid Discussions --- # Flipgrid Discussions: Feedback <img src="img/chat.jpg" width="720px" style="display: block; margin: auto;" /> --- class: inverse, center, middle # Research Tradeoffs <div class="padlet-embed" style="border:1px solid rgba(0,0,0,0.1);border-radius:2px;box-sizing:border-box;overflow:hidden;position:relative;width:100%;background:#F4F4F4"><p style="padding:0;margin:0"><iframe src="https://fsu.padlet.org/embed/2be7mvchzccn5l4n" frameborder="0" allow="camera;microphone;geolocation" style="width:100%;height:480px;display:block;padding:0;margin:0"></iframe></p><div style="padding:8px;text-align:right;margin:0;"><a href="https://padlet.com?ref=embed" style="padding:0;margin:0;border:none;display:block;line-height:1;height:16px" target="_blank"><img src="https://padlet.net/embeds/made_with_padlet.png" width="86" height="16" style="padding:0;margin:0;background:none;border:none;display:inline;box-shadow:none" alt="Made with Padlet"></a></div></div> --- # Self-Report Research <img src="img/streetlamp.jpg" width="480px" style="display: block; margin: auto;" /> - Humans are bad at estimating -- - (People are also bad at remembering) -- - People change once they know you're observing -- ### <svg aria-hidden="true" role="img" viewBox="0 0 448 512" style="height:1em;width:0.88em;vertical-align:-0.125em;margin-left:auto;margin-right:auto;font-size:inherit;fill:#782F40;overflow:visible;position:relative;"><path d="M190.5 66.9l22.2-22.2c9.4-9.4 24.6-9.4 33.9 0L441 239c9.4 9.4 9.4 24.6 0 33.9L246.6 467.3c-9.4 9.4-24.6 9.4-33.9 0l-22.2-22.2c-9.5-9.5-9.3-25 .4-34.3L311.4 296H24c-13.3 0-24-10.7-24-24v-32c0-13.3 10.7-24 24-24h287.4L190.9 101.2c-9.8-9.3-10-24.8-.4-34.3z"/></svg> Observer Effect --- # Unobtrusive Research <img src="img/telescope.jpg" width="450px" style="display: block; margin: auto;" /> -- - Digital trace data from social media -- - Reports and learning analytics from an LMS -- ### <svg aria-hidden="true" role="img" viewBox="0 0 448 512" style="height:1em;width:0.88em;vertical-align:-0.125em;margin-left:auto;margin-right:auto;font-size:inherit;fill:#782F40;overflow:visible;position:relative;"><path d="M190.5 66.9l22.2-22.2c9.4-9.4 24.6-9.4 33.9 0L441 239c9.4 9.4 9.4 24.6 0 33.9L246.6 467.3c-9.4 9.4-24.6 9.4-33.9 0l-22.2-22.2c-9.5-9.5-9.3-25 .4-34.3L311.4 296H24c-13.3 0-24-10.7-24-24v-32c0-13.3 10.7-24 24-24h287.4L190.9 101.2c-9.8-9.3-10-24.8-.4-34.3z"/></svg> Researcher Bias --- # Research Tradeoffs <img src="img/chocolate.jpg" width="540px" style="display: block; margin: auto;" /> -- - Breadth vs. Depth: -- - Trends in global chocolate consumption during the pandemic -- - "Describe a place that this chocolate reminds of..." --- class: inverse, center, middle # Module 2 readings --- class: inverse, center, middle # Evaluating Training as a System --- # Darabi (2002) <img src="img/system.jpg" width="600px" style="display: block; margin: auto;" /> -- - **Systems approach** to evaluation of training programs -- - systematic and procedural presentation of otherwise chaotic process -- - **whole picture** before and while considering individual pieces --- # Darabi & Arrington (2017) <img src="img/Fig1 (Darabi & Arrington, 2017).png" width="600px" style="display: block; margin: auto;" /> --- # Darabi & Arrington (2017) <img src="img/Fig2 (Darabi & Arrington, 2017).png" width="720px" style="display: block; margin: auto;" /> --- # Darabi & Arrington (2017) <img src="img/flowchart.jpg" width="560px" style="display: block; margin: auto;" /> ### Systemic Logic Model - Inputs - Processes - Outputs - Outcomes --- # Darabi & Arrington (2017) <img src="img/flowchart.jpg" width="560px" style="display: block; margin: auto;" /> ### Systemic Logic Model - Inputs <svg aria-hidden="true" role="img" viewBox="0 0 448 512" style="height:1em;width:0.88em;vertical-align:-0.125em;margin-left:auto;margin-right:auto;font-size:inherit;fill:#782F40;overflow:visible;position:relative;"><path d="M190.5 66.9l22.2-22.2c9.4-9.4 24.6-9.4 33.9 0L441 239c9.4 9.4 9.4 24.6 0 33.9L246.6 467.3c-9.4 9.4-24.6 9.4-33.9 0l-22.2-22.2c-9.5-9.5-9.3-25 .4-34.3L311.4 296H24c-13.3 0-24-10.7-24-24v-32c0-13.3 10.7-24 24-24h287.4L190.9 101.2c-9.8-9.3-10-24.8-.4-34.3z"/></svg> **Reaction** - Processes <svg aria-hidden="true" role="img" viewBox="0 0 448 512" style="height:1em;width:0.88em;vertical-align:-0.125em;margin-left:auto;margin-right:auto;font-size:inherit;fill:#782F40;overflow:visible;position:relative;"><path d="M190.5 66.9l22.2-22.2c9.4-9.4 24.6-9.4 33.9 0L441 239c9.4 9.4 9.4 24.6 0 33.9L246.6 467.3c-9.4 9.4-24.6 9.4-33.9 0l-22.2-22.2c-9.5-9.5-9.3-25 .4-34.3L311.4 296H24c-13.3 0-24-10.7-24-24v-32c0-13.3 10.7-24 24-24h287.4L190.9 101.2c-9.8-9.3-10-24.8-.4-34.3z"/></svg> **Learning** - Outputs <svg aria-hidden="true" role="img" viewBox="0 0 448 512" style="height:1em;width:0.88em;vertical-align:-0.125em;margin-left:auto;margin-right:auto;font-size:inherit;fill:#782F40;overflow:visible;position:relative;"><path d="M190.5 66.9l22.2-22.2c9.4-9.4 24.6-9.4 33.9 0L441 239c9.4 9.4 9.4 24.6 0 33.9L246.6 467.3c-9.4 9.4-24.6 9.4-33.9 0l-22.2-22.2c-9.5-9.5-9.3-25 .4-34.3L311.4 296H24c-13.3 0-24-10.7-24-24v-32c0-13.3 10.7-24 24-24h287.4L190.9 101.2c-9.8-9.3-10-24.8-.4-34.3z"/></svg> **Behavior** - Outcomes <svg aria-hidden="true" role="img" viewBox="0 0 448 512" style="height:1em;width:0.88em;vertical-align:-0.125em;margin-left:auto;margin-right:auto;font-size:inherit;fill:#782F40;overflow:visible;position:relative;"><path d="M190.5 66.9l22.2-22.2c9.4-9.4 24.6-9.4 33.9 0L441 239c9.4 9.4 9.4 24.6 0 33.9L246.6 467.3c-9.4 9.4-24.6 9.4-33.9 0l-22.2-22.2c-9.5-9.5-9.3-25 .4-34.3L311.4 296H24c-13.3 0-24-10.7-24-24v-32c0-13.3 10.7-24 24-24h287.4L190.9 101.2c-9.8-9.3-10-24.8-.4-34.3z"/></svg> **Results** --- class: inverse, center, middle # Evaluator Competencies --- # IBSTPI (2006) <img src="img/planning.jpg" width="480px" style="display: block; margin: auto;" /> ### Professional Foundations - Communicate effectively in visual, oral, and written form - Establish and maintain professional credibility - Demonstrate effective interpersonal skills - Observe ethical and legal standards - Demonstrate awareness of the politics of evaluation --- # IBSTPI (2006) <img src="img/planning.jpg" width="480px" style="display: block; margin: auto;" /> ### Planning & Designing the Evaluation - Develop an effective evaluation plan - Develop a management plan for the evaluation - Devise data collection strategies to support the evaluation questions and design - Pilot test the data collection instruments and procedures --- # IBSTPI (2006) <img src="img/planning.jpg" width="480px" style="display: block; margin: auto;" /> ### Implementing the Evaluation Plan - Collect data - Analyze and interpret data - Disseminate and follow-up the findings and recommendations --- # IBSTPI (2006) <img src="img/planning.jpg" width="480px" style="display: block; margin: auto;" /> ### Managing the Evaluation - Monitor the management plan - Work effectively with personnel and stakeholders --- # Tourmen (2009) <img src="img/planning.jpg" width="480px" style="display: block; margin: auto;" /> ### Making evaluation choices with experience - Method-oriented vs. results-oriented - Contextual/situational factors - Actively transforming the situation --- class: inverse, center, middle # Questions? <img src="img/question.jpg" width="480px" style="display: block; margin: auto;" /> **What questions can I answer for you now?** **How can I support you this week?** <hr> <svg aria-hidden="true" role="img" viewBox="0 0 512 512" style="height:1em;width:1em;vertical-align:-0.125em;margin-left:auto;margin-right:auto;font-size:inherit;fill:white;overflow:visible;position:relative;"><path d="M502.3 190.8c3.9-3.1 9.7-.2 9.7 4.7V400c0 26.5-21.5 48-48 48H48c-26.5 0-48-21.5-48-48V195.6c0-5 5.7-7.8 9.7-4.7 22.4 17.4 52.1 39.5 154.1 113.6 21.1 15.4 56.7 47.8 92.2 47.6 35.7.3 72-32.8 92.3-47.6 102-74.1 131.6-96.3 154-113.7zM256 320c23.2.4 56.6-29.2 73.4-41.4 132.7-96.3 142.8-104.7 173.4-128.7 5.8-4.5 9.2-11.5 9.2-18.9v-19c0-26.5-21.5-48-48-48H48C21.5 64 0 85.5 0 112v19c0 7.4 3.4 14.3 9.2 18.9 30.6 23.9 40.7 32.4 173.4 128.7 16.8 12.2 50.2 41.8 73.4 41.4z"/></svg> [bret.staudtwillet@fsu.edu](mailto:bret.staudtwillet@fsu.edu) | <svg aria-hidden="true" role="img" viewBox="0 0 512 512" style="height:1em;width:1em;vertical-align:-0.125em;margin-left:auto;margin-right:auto;font-size:inherit;fill:white;overflow:visible;position:relative;"><path d="M459.37 151.716c.325 4.548.325 9.097.325 13.645 0 138.72-105.583 298.558-298.558 298.558-59.452 0-114.68-17.219-161.137-47.106 8.447.974 16.568 1.299 25.34 1.299 49.055 0 94.213-16.568 130.274-44.832-46.132-.975-84.792-31.188-98.112-72.772 6.498.974 12.995 1.624 19.818 1.624 9.421 0 18.843-1.3 27.614-3.573-48.081-9.747-84.143-51.98-84.143-102.985v-1.299c13.969 7.797 30.214 12.67 47.431 13.319-28.264-18.843-46.781-51.005-46.781-87.391 0-19.492 5.197-37.36 14.294-52.954 51.655 63.675 129.3 105.258 216.365 109.807-1.624-7.797-2.599-15.918-2.599-24.04 0-57.828 46.782-104.934 104.934-104.934 30.213 0 57.502 12.67 76.67 33.137 23.715-4.548 46.456-13.32 66.599-25.34-7.798 24.366-24.366 44.833-46.132 57.827 21.117-2.273 41.584-8.122 60.426-16.243-14.292 20.791-32.161 39.308-52.628 54.253z"/></svg> [@bretsw](https://twitter.com/bretsw) | <svg aria-hidden="true" role="img" viewBox="0 0 496 512" style="height:1em;width:0.97em;vertical-align:-0.125em;margin-left:auto;margin-right:auto;font-size:inherit;fill:white;overflow:visible;position:relative;"><path d="M336.5 160C322 70.7 287.8 8 248 8s-74 62.7-88.5 152h177zM152 256c0 22.2 1.2 43.5 3.3 64h185.3c2.1-20.5 3.3-41.8 3.3-64s-1.2-43.5-3.3-64H155.3c-2.1 20.5-3.3 41.8-3.3 64zm324.7-96c-28.6-67.9-86.5-120.4-158-141.6 24.4 33.8 41.2 84.7 50 141.6h108zM177.2 18.4C105.8 39.6 47.8 92.1 19.3 160h108c8.7-56.9 25.5-107.8 49.9-141.6zM487.4 192H372.7c2.1 21 3.3 42.5 3.3 64s-1.2 43-3.3 64h114.6c5.5-20.5 8.6-41.8 8.6-64s-3.1-43.5-8.5-64zM120 256c0-21.5 1.2-43 3.3-64H8.6C3.2 212.5 0 233.8 0 256s3.2 43.5 8.6 64h114.6c-2-21-3.2-42.5-3.2-64zm39.5 96c14.5 89.3 48.7 152 88.5 152s74-62.7 88.5-152h-177zm159.3 141.6c71.4-21.2 129.4-73.7 158-141.6h-108c-8.8 56.9-25.6 107.8-50 141.6zM19.3 352c28.6 67.9 86.5 120.4 158 141.6-24.4-33.8-41.2-84.7-50-141.6h-108z"/></svg> [bretsw.com](http://bretsw.com)