Deliver Written document (Portable Document Format) Submit PDF via Blackboard
Rationale and Description
Evidence-based practice is an important aspect of Decision Science. In this assessment, you will have the chance to create and run your own study on human participants via crowdsourcing.
This assessment will involve researching human decision making by developing a crowdsourced research design, running the research design and writing a report based on the findings. This will demonstrate your ability to understand some of the relevant literature in the area, identify a method to learn more about how humans make decisions through experimentation, as well as critically analyse human data.
You will use your knowledge and skills gained from the workshops and materials presented in the course, and apply them to a research problem. You will use your unique knowledge from the data you collect to provide a research report explaining what you have found, as well as suggest current or future technological innovations which may implement your findings.
Learning Outcomes
A successful completion of this task will demonstrate:
1. An ability to form a clear and answerable hypothesis associated with a given research
topic, and justify this with a succinct analysis of relevant literature.
2. An ability to formulate an effective experimental protocol to answer the hypothesis.
3. An understanding of how experimentation can inform future innovations in decision
support technologies
4. An understanding of how humans make decisions while interacting with information.
5. An ability to analyse experimental data in order to address a research topic.
6. An ability to effectively communicate the report with appropriate written expression.
7. An understanding of requirements to conduct a study according to ethical guidelines.
8. An ability to work effectively in a group and manage equal contribution across team members
Essential Elements
You must submit a research report with the following sections:
1. Title
2. Abstract
3. Introduction
4. Method
5. Results
6. Discussion
7. References
8. Statement of Contribution
You must write the report as a team, and will be asked to provide evidence of this in the form of a statement of contribution. In writing your report, you must convince the reader that:
1. You have identified a justifiable hypothesis to address your research topic;
2. You have developed a feasible, practical methodology for testing the hypothesis, using
contemporary tools and techniques and reasonable resources;
3. You understand your results and effectively communicate them in the context of the
research problem; and
4. Your study adds value by identifying novel implications.
In order to convince the reader of these points your argumentation must be:
• Clear — Your writing must be easily understandable by a lay reader, avoiding uncommon terminology and abbreviations.
• Concise — You must express your ideas efficiently, so that key points are not obscured by irrelevant material.
• Coherent — Your arguments and the conclusions you draw must be structured logically. • Convincing — The overall “story” you tell must be compelling and believable.
Information & links to resources to help you with writing research reports will be provided in the ‘detailed instructions’ section below.
Ethics
Please note that, as your research projects use human subjects, they are subject to ethical guidelines for human research. Research projects conducted in this unit are covered by ethics approval number 1900000132. As such, the guidelines set by this approval must be followed, and any divergence from this will be considered as a breach. The teaching team’s approval of your methodology is therefore essential before commencing data collection. The teaching staff will ascertain whether your proposed methodology meets ethics requirements and will approve or reject it. If you commence any form of data collection from human subjects (whether on the prolific or any other platform, through social media, friends, etc.) without having your methodology approved by teaching staff, this will result in an automatic fail for this assignment.
Marking Criteria
This assessment is criteria referenced, meaning that your grade for the assessment will be given based on your ability to satisfy key criteria. Refer to the attached ‘Criteria Sheet’ and ensure you understand the detailed criteria.
It is important to realise that the assessment does not merely require that you know or understand the material, but that you are able to effectively demonstrate and provide evidence of your understanding to the marker through the report you produce. This means that you need to ensure you are making your knowledge and understanding clear to the person marking your assignment.
You will not receive marks or percentages for this assessment. You will receive an overall grade (e.g. pass – 4, high distinction – 7) based on the extent to which you meet the criteria. The weighting of each criteria is provided in the criteria sheet.
Feedback
Essential step (contributing to Part A): A hypothesis, summary of your review of the literature to justify this hypothesis, and completed research design will be required for presentation to teaching staff by the end of the tutorial in week 9. This will enable teaching staff to provide formative feedback, enabling you to:
(a) understand where you can make improvements to ensure your study will pass the requirements for Ethics Approval, and
(b) spread your assignment load more evenly across the semester.
Note: whilst this formative feedback does not count towards your final grade, it is highly recommended in order to ensure your design passes the Ethics Approval stage (Due: End of Week 9) – you will not be approved to collect your data unless you have passed the ethics requirement.
If the methodology you submit at the end of week 9 does not pass the Ethics Approval, you will be provided with a pre-designed study with pre-collected data and your grade for this assignment will be capped at a credit (5).
Groupwork
Note: this assignment is group-based (pairs of 2), and as such, participating in a group is an essential component. You are responsible for forming teams and this will need to be done early (in Week 7), and need to be finalised by 9am Thursday 5th May at the latest.
Teaching staff will assist in facilitating the initial stages of group formation, but ultimately, this is your responsibility. If you are struggling to find a group, please use Slack to find others who do not yet have a group, and ensure you do this well ahead of the deadline for formative feedback. If you have not formed a group, or have not contributed meaningfully to your group, by the due date for the presentation of your draft introduction and completed methodology (Part A), you will not receive approval to go ahead with your study. You will therefore be provided with a pre-designed study with pre-collected data and your grade for this will be capped at a credit (5).
Detailed Instructions
To assist you in planning ahead to ensure you meet essential deadlines, we have broken the assignment into the following shorter steps.
Part A – Weeks 7-9
Step 1 – develop preliminary hypothesis In week 7’s workshop, you will finalise team formation and develop a preliminary hypothesis. In Week 8’s workshop, this hypothesis will be further developed into a methodology.
Step 2 – begin literature review You will then conduct a brief review of literature to inform and justify your hypothesis. Note that, upon completing this step, you may need to alter your preliminary hypothesis and/or methodology to align it with relevant literature.
In order to restrict your literature review to only the most relevant sources to justify your methodology and hypothesis, you are only required to select 6-8 references to address the following question:
What evidence exists to justify why I would expect this [hypothesis] to happen?
Step 3 – design experiment (with ethics documents) In week 8’s workshop, you will develop your experimental design to test your hypothesis. The next step is to create the experiment, which will take the form of either one or two Google forms (depending on whether you have 2 or 1 condition). This will be what will be provided to participants to collect the experimental data. For a guide on creating and sharing a google form for your experiment, videos and reference material will be provided via Slack and Blackboard.
This, along with completed ethics documents that will be made available on Blackboard, will be presented in consultation by week 9 at the latest and will form part of your formative assessment.
Step 4 – present hypothesis, justification, experiment, and ethics documents to teaching team for feedback Create a presentation, in 2 slides, of your hypothesis, and a summary of the evidence you will be citing to justify your hypothesis.
Present your slides as well as your completed Google Form(/s) and Ethics forms to teaching staff in the Week 9 tutorial. Note, you will have around 7 minutes for this, so you need to be prepared and succinct in your presentation in order to receive the most helpful feedback.
If you would like feedback about your methodology earlier than week 9, you can contact the teaching staff on Slack or receive face-to-face feedback during the workshop times in weeks 7-9. The latest date for presentation to teaching staff for feedback on hypothesis, research design and ethics documents is the workshop in week 9.
Step 5 – submit experiment and ethics documents for final approval Integrate feedback provided during your presentation to teaching staff and make necessary edits to you design and/or ethics forms, and submit via Blackboard for final Ethics Approval by Friday 13th May. This is an essential element in order for the teaching team to approve your experimental methodology to be put on the crowdsourcing platform to collect human data [step 6]. Note that, this is the final chance to have your design approved. If it is not approved, we will provide you with a pre-designed study with results that you can use to complete your assignment. In this case, the maximum grade that can be achieved for this assignment is a credit (5).
Final submission for Part A DUE: Friday 13th May
Part B – Weeks 10-12
Step 6 – place your study on Prolific Once Part A is complete and if your study has been approved, you can then place your experiment on the crowdsourcing platform, Prolific. In order to do this, you will be given a username and password for a Prolific account with a certain amount of funds loaded to be used to pay participants.
It is expected that you access the guide provided on blackboard for placing experiments on Prolific and follow this, asking any questions via slack or email if you are unsure of any part of the process.
Step 7 – analyse data Once you have collected your data, we will be providing learning resources to help you determine which analysis method to use. We will not give you the answers, but will help you know which questions to ask to determine which analysis to use.
Step 8 – write your report The final step involves writing a report to communicate your findings. This is the formally assessable component, and requires all previous steps to be conducted. The required sections of the report are as follows.
Written Report – Essential Requirements:
1. Title. Come up with an concise, informative title for your study, and list the authors (team members) in alphabetical order with students numbers.
2. Abstract. This is a half-page summary of your entire report. It should follow the following structure:
• What you did
• Why you did it
• How you did it
• What you found
3. Introduction. This will be one to one and a half (1 – 1.5) pages long. You are required to provide a justification of why you think the results will turn out the way you have formulated in the hypothesis.
• Short background leading to why you would expect humans to behave a certain way, given your research topic
i. Use 6-8 relevant references/citations to justify your hypothesis* • State the research topic and hypothesis of your study
4. Methodology. This describes your experiment in enough detail for another researcher to replicate it, and includes:
• Summary of participants (e.g. describe who participants were)
• Materials used (e.g. describe what participants saw)
• Experimental procedure (e.g. describe what participants did)
5. Results. This describes the results of your experiment, and includes:
• Summary of data in table/graphical form
• Analysis method used – how you analysed the data
• Outline of results from data analysis
6. Discussion. This will be one to one and a half (1 – 1.5) pages long. This places your results back into the context you gave in your introduction, and demonstrates what your study has found, including:
• Whether the results confirm or disconfirm your hypothesis and what that implies for your research topic
• How these results relate to your previously referenced sources (from introduction)
• Limitations of your study
• Implications of your study for decision support technologies (e.g. search
engines, recommendation systems, dashboards, etc.)
7. References. This is the list of all literature referenced in the body of your report. All
references should be in APA format both in the body of the report and in the reference
list.
8. Statement of Contribution. You will agree on your degree of contribution, which you will
include in writing at the end of your document. This will be in the following format:
• Experimental design: Student 1 (%), Student 2 (%); Analysis: Student 1 (%),
Student 2 (%); Introduction: Student 1 (%) and Student 2 (%); Methodology: Student 1 (%), Student 2 (%); Results: Student 1 (%), Student 2 (%); Discussion: Student 1 (%), Student 2 (%)
9. Appendix. In this section, provide the table(/s) of your raw results.
* Note that these references must be cited in the body of your introduction and listed in a reference list, rather than just being added as a bibliography/reading list at the end.
For information on how to write high quality reports, please see the following links (note, however, that we do not require you to write a complete introduction section, so please continue to refer back to the above for the essential requirements):
• https://www.citewrite.qut.edu.au/write/empiricalarticle.jsp
• http://www.discoveringstatistics.com/docs/writinglabreports.pdf
Clarification
If you require any clarification on the requirements of the assessment, as outlined in this document, please ensure you speak to teaching staff during workshop times or via Slack, as soon as possible to ensure you do not fall behind.
Please note, this clarification applies to assessment requirements as well as general questions about logistics involved in running crowdsourced studies. We cannot provide any specific feedback on your research or provide you with specific help which may provide you with an unfair advantage. In addition, if you need any help with writing, please see HiQ for assistance.
Resources
In addition to the links provided above and the preparatory materials and workshops included in this course, we will provide a “Crowdsource Guide”, which can be accessed on Blackboard, and consists of a text-based and video-based guide to crowdsourcing. We will also provide a mandatory ethics training video that you will need to watch before being approved to begin collection of data from human participants. Please ensure you read and watch these resources before asking any questions about how to conduct crowdsourcing for this assignment.
Questions
Questions relating to the assessment should be directed initially to the teaching staff during the workshops and consultation times. The teaching staff may address these for the benefit of the whole class.
If you need to clarify something outside of these times, please direct your queries to the Slack team. A fellow student may have the same question and may benefit from the answer. Additionally, another student may know the answer to your question and be able to answer it at times when the teaching staff are unavailable (i.e. outside business hours). Teaching staff will monitor and respond to queries on the Slack channel, however, will only be able to do this during normal business hours.
Criteria Sheet – Assessment 2 Experimental Presentation of a Theory – IFN521 Foundations of Decision Science
7 – High Distinction
6 – Distinction
5 – Credit
4 – Pass
3 – Marginal Fail
2 – Fail
1 – Low Fail
Use of literature as supporting evidence (25% of total grade) – The ability to:
• draw on authoritative sources as needed to support a hypothesis and methodology (and demonstrate research capability by finding the majority of sources outside the unit set reading list)
• analyse, critique and evaluate literature sources
• understand the significance of the literature to the topic
7
6
5
4
3
2
1
• Citations are provided to support all significant points in the argument.
• Hypothesis and methodology are very clearly and logically informed by the citations and argument provided • All sources are consistently relevant (to the research topic), current and authoritative. • Referencing is thorough and consistently accurate.
• Excellent level of critical analysis
• Citations are provided to support almost all significant points
in the argument.
• Hypothesis and methodology are clearly and logically informed by the citations and argument provided
• Sources are mostly relevant (to the research topic), current and authoritative.
• One or two insignificant errors apparent in the referencing.
• High level of critical analysis
• Citations are provided to support all but one or two significant points in the argument.
• Hypothesis and methodology are mostly clearly and logically informed by the citations and argument provided • Sources are mostly relevant (to the research topic), current and authoritative.
• Referencing is slightly inconsistent or contains a few minor errors.
• Sound level of critical analysis
• Citations are provided, but a few significant points in the argument lack supporting evidence. • Hypothesis and methodology are somewhat clearly and logically informed by the citations and argument provided
• Half the sources are relevant (to the research topic), current and authoritative.
• Several noticeable referencing errors.
• Satisfactory level of critical analysis
• Limited use of citations to support key parts of the argument.
• It is substantially unclear how hypothesis and methodology has been informed by the citations and argument provided
• A minority of sources are relevant (to the research topic) but may not be authoritative.
• Some significant referencing errors. • Some evidence of critical analysis
• Inadequate citations to support the bulk of the argument.
• There is scant connection between the argument and citations used and the hypothesis and methodology
• Sources are mostly irrelevant and/or not authoritative.
• Many significant referencing errors.
• Little evidence of critical analysis
• No attempt to use the literature to support argument
• The hypothesis and methodology do not appear to have been informed by the citations and argument provided • Literature cited is irrelevant to the arguments.
• Referencing is absent or erratic.
• No evidence of critical analysis
Continue to next page…
Hypothesis, Experimental Design and Ethics (15% of total grade) – Ability to:
• Specify a well-formed hypothesis that is a precise, testable hypothesis appropriately expressed in terms of independent and dependent variables relevant to information- based decisions
• Design a feasible, stepwise research methodology to appropriately address the hypothesis
• Demonstrate understanding of Human Research Ethics guidelines
7
6
5
4
3
2
1
• Hypothesis is excellently formed and extremely relevant to information-based decisions
• It is made obvious that completing the process will contribute to testing the hypothesis.
• Design adheres to Human Research Ethics Guidelines
• Hypothesis is very well formed and very relevant to information-based decisions
• It is made very clear that completing the process will contribute to testing the hypothesis.
• Design adheres to Human Research Ethics Guidelines
• Hypothesis is well formed and relevant to information-based decisions
• It is made clear that completing the process will contribute to testing the hypothesis.
• Design adheres to Human Research Ethics Guidelines
• Hypothesis is satisfactorily formed and reasonably relevant to information-based decisions
• It is generally clear that completing the process will contribute to testing the hypothesis.
• Design adheres to Human Research Ethics Guidelines
• Hypothesis is not well formed and lacks relevance to information- based decisions
• It is not entirely clear how completing the process will contribute to testing the hypothesis.
• Design does not adhere to Human Research Ethics Guidelines
• Hypothesis is unsatisfactory and largely irrelevant to information- based decisions
• It is not at all clear that completing the process will contribute to testing the hypothesis.
• Design does not adhere to Human Research Ethics Guidelines
• Hypothesis is either missing or totally unsatisfactory
• There is no convincing argument that completing the process will contribute to testing the hypothesis.
• Design does not adhere to Human Research Ethics Guidelines
Research Methodology (10% of total grade) – Ability to:
• Describe a feasible, stepwise research methodology
• Identify the data collection and analysis steps needed to complete the process
• Estimate the resources needed to complete the project
7
6
5
4
3
2
1
• A comprehensive summary of participants is presented
• A comprehensive description of materials is presented
• The proposed research procedure is described extremely clearly, as a series of precise steps.
• It is made obvious that completing the process will contribute to testing the hypothesis.
• A mostly comprehensive summary of participants is presented
• A mostly comprehensive description of materials is presented
• The proposed research procedure is described very clearly, as a series of clear steps.
• It is made very clear that completing the process will contribute to testing the hypothesis.
• A clear summary of participants is presented, although lacks some information
• A clear description of materials is presented
• The proposed research procedure is described clearly, as a series of generally well-described steps.
• It is made clear that completing the process will contribute to testing the hypothesis.
• A summary of participants is presented, although lacks some clarity and information
• A description of materials is presented, however, some clarity and information is lacking • The proposed research procedure is generally clear, but some steps need further explanation. • It is generally clear that completing the process will contribute to testing the hypothesis.
• Participants involved in the study are not presented in enough detail and/or with enough clarity
• Materials are not presented in enough detail and/or with enough clarity
• The proposed research procedure is described weakly, or some key steps are unclear.
• It is not entirely clear how completing the process will contribute to testing the hypothesis.
• Little detail is given about participants involved in the study
• Little detail is given about materials used in study
• The proposed research procedure is described poorly and/or several key steps are described inadequately.
• It is not at all clear that completing the process will contribute to testing the hypothesis.
• No useful detail is given about participants
• No useful detail is given about materials used in study
• The proposed research procedure is inadequately described. • There is no convincing argument that completing the process will contribute to testing the hypothesis.
Analysis & Results (12% of total grade) – The ability to:
• Identify an appropriate analysis method
• Effectively execute the analysis method
• Appropriately present results
7
6
5
4
3
2
1
• Analysis method used is appropriate for the methodology
• Analysis is fully correct • Excellent presentation of results with complete detail
• Analysis method used is appropriate for the methodology, although a more appropriate method could have been used
• Analysis is largely correct
• Very good presentation of results with complete detail
• Analysis method used is mostly appropriate for the methodology
• Analysis is basically correct
• Good presentation of results with some minor details missing
• Analysis method used is somewhat appropriate for the methodology
• Analysis is satisfactory but with some non-trivial errors
• Adequate presentation of results with details missing
• Analysis method used is applicable but not appropriate for the methodology
• Analysis has some significant errors
• Barely satisfactory presentation of results with major detail missing
• Analysis method used is both inapplicable and inappropriate for the methodology
• Analysis is poorly executed
• Presentation of results is unsatisfactory
• No coherent analysis method is used
• No analysis is provided • Presentation of results is incomprehensible
Discussion (30% of total grade) – The ability to:
• Relate the results to the research topic (RT) & hypothesis
• Discuss the results in the context of previously referenced studies, as appropriate
• Identify the limitations of the study
• Justify the implications of the study
7
6
5
4
3
2
1
• Discussion very clearly states how the results address the hypothesis and relate to the RT
• Excellent critical discussion of results in the context of previously referenced studies
• Significant limitations of the study are very clearly communicated
• Discussion clearly states how the results address the hypothesis and relate to the RT
• Good critical discussion of results in the context of previously referenced studies
• Significant limitations of the study are clearly communicated
• Discussion creditably states how the results address the hypothesis and relate to the RT
• Well-written discussion of results in the context of previously referenced studies, however, little evidence of critical thought
• Discussion satisfactorily states how the results address the hypothesis and relate to the RT
• Adequate discussion of results in the context of previously referenced studies, however, little evidence of critical thought
• Connection made between results and hypothesis and RT is limited
• Limited discussion of results in the context of previously referenced studies, with little to no evidence of critical thought
• Connection made between results and hypothesis RT is mostly unclear
• Limited discussion of results in the context of previously referenced studies, however it is not clear and with little to no evidence of critical thought
• Results are not related to the hypothesis and RT • No discussion of results in the context of previously referenced studies is attempted
• No clear limitations are communicated
• Little to no clear or relevant implications of the study are presented.
• Implications of the study are excellently communicated – creating a very clear and logical picture of the usefulness of the results, with a very high degree of novelty
• Implications of the study are very well communicated – creating a clear and logical picture of the usefulness of the results, with a high degree of novelty
• Significant limitations of the study are clearly communicated, although some obvious limitations are not addressed
• Implications of the study are well communicated, and the usefulness of results is made clear, with a creditable degree of novelty
• Some limitations of the study are communicated with adequate clarity, although some obvious limitations are not addressed
• Implications of the study are adequately communicated, and the usefulness of results is made somewhat clear, with a passable degree of novelty
• Some limitations of the study are communicated with some clarity, although most obvious limitations are not addressed
• Implications of the study are somewhat communicated; however, their usefulness is lacking clarity and novelty
• Attempted communication of limitations that are irrelevant and/or unclear • Implications of the study are somewhat communicated; however, their usefulness is not made clear, and with no discernible novelty
Continue to next page…
Expression & Presentation (8% of total grade) – The ability to:
• use fluent language with correct grammar, spelling and punctuation
• use appropriate paragraph and sentence structures
• use appropriate style and tone of writing
• produce a professionally presented document
7
6
5
4
3
2
1
• Inclusion of all essential components of the report (Title, Abstract, Introduction, Methodology, Results, Discussion, References), with excellent document structure.
• Very clear yet concise writing throughout.
• Perfect use of standard grammar, spelling and
• Inclusion of all essential sections of the report (Title, Abstract, Introduction, Methodology, Results Discussion, References), with very good document structure.
• Clear yet
concise writing throughout.
• Grammar, spelling and
• Inclusion of all essential sections of the report (Title, Abstract, Introduction, Methodology, Results, Discussion, References) with good document structure.
• Clear yet
concise writing in most parts.
• Grammar, spelling and
• One missing essential section of the report (Title, Abstract, Introduction, Methodology, Results, Discussion, References), with generally good document structure.
• Clear yet concise writing in general but with a few unclear passages. • Grammar and/or
• Multiple missing essential sections of the report (Title, Abstract, Introduction, Methodology, Results, Discussion, References), and poor document structure.
• Several parts of the document are either too brief and unclear or contain significant
• Multiple missing essential sections of the report (Title, Abstract, Introduction, Methodology, Results, Discussion, References) and very poor document structure.
• Most parts of the document are either too brief and unclear or contain significant
• Essential sections (Title, Abstract, Introduction, Methodology, Results, Discussion, References) not present or not clearly outlined, and extremely poor document structure.
• Document is entirely unclear either because it is too brief and missing important points or
punctuation.
• Polished professional appearance.
punctuation mainly accurate.
• Professional presentation.
punctuation creditably accurate.
• Neat and tidy presentation.
spelling and/or punctuation are satisfactory.
• Unprofessional, untidy, or unattractive presentation in a few places.
amounts of irrelevant material.
• Grammar and/or spelling and/or punctuation contain significant errors.
• Unprofessional, untidy, or unattractive presentation in many places.
amounts of irrelevant material.
• Grammar, spelling and punctuation contain numerous and distracting errors.
• Unprofessional, untidy, or unattractive presentation in most places.
consists of large amounts of irrelevant material.
• Meaning unclear throughout due to major errors in grammar, spelling and punctuation. • Disorganised or incoherent writing.
Looking for answers ?
