Reliability (DP IB Psychology): Revision Note

Reliability

• Reliability is achieved if a study has been set up so that the IV can be seen to affect the DV

  • If the procedure is replicated, the study should show similar results

  • Replicating a study and finding similar results shows that the measure is consistent and not liable to fluctuate to any significant degree

• If a study is completed using a standardised procedure and obtains the same results, it can be said that it is reliable 

Reliability across methods

• Lab experiments are the most reliable method, as they:

  • take place in a neutral space under controlled conditions

  • follow a standardised procedure

  • use random allocation of participants to conditions

  • tend to use a control group as a comparison to the experimental group

  • generate quantitative data which is easy to compare and analyse

• Field experiments generate quantitative data and manipulate an IV but they are exposed to extraneous variables that reduce reliability

• Natural experiments generate quantitative data but they use a naturally occurring IV over which the research has no control, making them less reliable than lab experiments

Types of reliability

• Internal reliability: The extent to which a measure is consistent within itself

  • E.g., all items in a questionnaire measuring stress should give consistent results

  External reliability: The extent to which a measure is consistent over time

  • E.g., a test should produce similar results if given to the same person at two different times

Measuring reliability

• The test-retest method measures external reliability: 

  • The same participants are given the same questionnaire at separate time intervals (e.g., with a 6-month gap between testing sessions) 

  • If scores are consistent, external reliability is established

• The split-half method measures internal reliability:

  • The researcher splits the test in half and analyses the responses given to the first half of the questionnaire compared to the second half of the questionnaire

  • If responses are similar across both halves, internal reliability is established

Inter-observer reliability

• For an observational study, the researchers will have previously agreed on which specific behaviours should be recorded so that all observers are in agreement before the observation 

• Researchers can test the reliability of their observations by comparing them with another researcher's recording of their behaviours

  • The level of consistency between the two records is then compared

• Inter-observer reliability is the level of consistency between two or more trained observers when they conduct the same observation, as follows:

  • Observers agree in advance on behaviour categories and how to record them

  • Each observer records behaviours independently to avoid conformity

  • After the observation, records are compared (often using a tally chart)

  • A correlation is calculated between observers’ data

    • A strong positive correlation indicates good inter-observer reliability

• Establishing good inter-observer reliability means that there is less chance that researcher bias has interfered with the observation

An example of a tally chart used to record categories of behaviour in an observation: