Virtual Advisory Service (Pilot) Phase Two Evaluation

The Virtual Advisory Service is an online virtual community which allows teachers to access information, guidance and advice on the implementation of the curriculum. An initial pilot was carried out in 2007 with four local authority areas covering the subjects of science, numeracy and literacy. The evaluation of this phase indicated that the service was well received, and following this feedback the service was further developed.

A second pilot phase was run from November 2007 to June 2008. This phase was carried out with twelve local authorities and was extended to include the following subjects expressive arts, health and wellbeing, languages, mathematics, religious and moral education, science, social studies and technologies, as well as, science, numeracy and literacy. Learning and Teaching Scotland commissioned George Street Research to evaluate pilot phase two and this work was completed in August 2008.

Evaluation aim

The overall aim of the evaluation was to determine the extent to which the project aim and objectives were met during pilot phase two and to develop a series of recommendations to assist the future development of the Virtual Advisory Service.

Research method

A wide range of research techniques were used during the evaluation. Phase 1 of the evaluation involved secondary desk research to assess existing models of online communities for educational practitioners. Running concurrently with this was an online quantitative survey with registered users of the service; 128 responses were collected.

Phase 2 of the project comprised a second online quantitative survey with users of the service (using a very similar questionnaire to that used in phase 1; 88 responses were collected). Registered users from five local authorities were selected to take part in a series of four qualitative focus groups. Semi-structured, telephone interviews were conducted with 41 Quality Improvement Officers and Directors of Education, representing 31 local authorities in Scotland. 15 in-depth telephone interviews were conducted with individuals involved in the Virtual Advisory Service; four with Steering Group members, four with Virtual Advisers and seven with non-engagers of the service.

Key findings

Overall, Pilot Phase Two of the Virtual Advisory Service was successful in many respects, despite falling slightly short of some of the overall project objectives.
Around seven in 10 users (71%) were satisfied overall with the Virtual Advisory Service, against a project aim of 75%. Although this result was just short of the project target, this should not be viewed as a discouraging finding. The results showed that those logging onto the service most frequently were more satisfied than those logging on infrequently.
Qualitative feedback showed that some initial sceptics towards the service changed their minds after having a positive experience with an answered enquiry. This would suggest that having the service available to all practitioners is still useful, even for those who do not currently perceive a need for it as their circumstances may change.
One of the successes of the Virtual Advisory Service so far is that for a small number of users, within a relatively short space of time, there has already been a positive impact in the classroom. Around half (48%) were introduced to new teaching practices and users seem optimistic in the extent to which the service will provide assistance in the future, especially in the delivery of Curriculum for Excellence.

Recommendations

There are two key areas of recommendations:

1. User engagement/communication

An initial consideration which must be addressed is the extent to which some non-engagement with the service is expected and accepted.
The use of generic email is unlikely to be the most effective medium of recruitment or initial communication; respondents did not prioritise emails.
It is vitally important that a clear role of local authority co-ordinators is developed to ensure clear communication and to ‘legitimise’ the importance of VAS.

2. Service features

Regular updates to the service are likely to increase user integration. Attention should be drawn to these features with strong sectioning and colouring.
Users should be able to register for updates when there is some activity relating to specific subjects / areas of interest.
It is worth considering empowering users to self-select enquiry response times –thus urgent enquiries can be prioritised over those of less immediate importance.
Respondents value the option to send anonymous enquiries, but this should not be mandatory.
There needs to be a facility in place to discuss answered enquiries publicly; through automatically generated discussion forums, or a Discuss This link.
The expansion of VAS into a service which deals with all areas of teaching would be very beneficial; thus moving away from subjects alone.

Full report

Please contact the Research team (research@LTScotland.org.uk) for copies of the full report.