An Overview of the Co-Inform Webinar Based Workshops for remote UX Testing of Tools at the Time of COVID-19

Several people had the opportunity to evaluate Co-Inform’s plugin in Stockholm, Athens and Thessaloniki. The workshops were held online and this meant that a series of mini-workshops needed to take place in each country, to reach a sufficient number of participants for research evaluation purposes. Only a small group of people, 2 to 3 participants, can participate at a time.

A research team from Stockholm University, Open University and The Cyprus University of Technology co-designed the questionnaires for evaluating the plugin. The objective of the questionnaire is to explore how much the participants understand and trust the tool that they are presented with during the session.

eGovlab at Stockholm University evaluated the plugin with 5 citizens in early June 2020. As a second step, SU eGovlab held a training session with the International Hellenic University and the and the International Institute for Applied Systems Analysis, to transfer the toolkit and guidelines of the workshop process. IHU proceeded by successfully conducting a workshop with 2 participants and more sessions are scheduled to take place soon.

Scytl has been offering technical support during the online sessions. 

Methodology (Webinar Based Workshop key activities)

As first step moderators and participants meet and greet in a setup video-conference space. For the workshop’s purposes, participants share their screens for moderators to be able to record the screen actions in order to later analyse the behaviour in interacting with the plugin.

Session 1

• Co-inform plugin installation and evaluation

As a second step participants install the plugin sent to them by following installation instructions. After installing the plugin, participants open their Twitter feed which now is synched in with the Co-inform plugin.

There is then a five-minute window for participants to freely navigate in their Twitter and explore how Co-inform’s plugin evaluates different tweets (e.g. credible, not credible, etc.).

After a while, moderators ask participants to perform certain tasks and then rate their experience in an online questionnaire as they go.

• Fill-in Questionnaire

Session 2 – Open Questions session

After having performed all the tasks and answered the questions, an open discussion between moderators and participants takes place. The questions are now open-ended and allow more room for participants to express their thoughts vis-a-vis the tool that they tested.

Discussion  and Findings

Pros and Cons

During the workshop sessions, we observed a diverse set of pros and cons of remote UX testing sessions. In the forthcoming paragraph, we explained a few of them, along with related valuable information. We noted that some items are more straightforward remotely, and some things are more comfortable in the face-to-face meet-up. Thanks to smart technology solutions that facilitate us to organize remote UX testing in the time of COVID-19, while in some co-inform piloting countries, public administrations have already issued special public notices to avoid public gathering.

Pros:

• The co-inform piloting countries can get rapid digital outcomes from the 3^rd webinar-based workshops that are cool to process and share with the co-inform development team. The teams can observe what the users are doing during the UX testing sessions.
• The participants are already familiar with the technology that is used for the webinar-based workshops. Easy to set up such technology. The piloting countries get a recording, and it is simple to further use in co-inform R&D work as well.

Cons:

• Lack of stable internet access at participants’ end is noted as a critical barrier during the co-inform workshop sessions.
• The co-inform participants are somehow tech-savvy. However, they are performing a UX testing of fact-checking tools that are in the under-development phase.
• In the workshop sessions, the moderators are dependent on the participants’ capability to operate the tools.

Recommendations for Future Remote UX Testing Sessions

• The co-inform piloting countries moderators need to be planned for the webinar-based workshops for remote UX testing.
• The moderator should be aware of the tools to be tested in terms of its functionalities, loopholes, and tools scope domain like misinformation.
• The availability of stable internet access along with its backup facility, like data service on personal mobile, maybe ensured at both ends for smooth sessions about remote UX testing.
• The moderator and participants should set up and test their desktop, laptops, and other associated computer accessories that include microphones, speakers, cameras, and communication software tools like Zoom.

Conclusion

The third round of workshops across Greece, Austria and Sweden have kicked start in June 2020. Citizens and policymakers are evaluating two co-inform tools that include a co-inform plugin and co-inform dashboard. The co-inform dashboard is built-on the co-inform plugin. This time around the workshops are online, due to the COVID-19 pandemic. This has not been an obstacle in gathering requirements and feedback on usability. The goal of the webinar-based workshops is to evaluate the co-inform tools in terms of its usability by the stakeholders (citizens, journalists, and policymakers) and obtain their feedback to improve the functioning of tools.

It is the first time that stakeholders are performing live testing of the co-inform tools. In the preceding two co-inform workshops, participants had discussed the possible functionalities of the co-inform tools and provided feedback on the paper-based wireframes (static images) of the tools. It is a major step for the project, which will inform both the research and development workstreams.

In this article, we shared the workshop experiences for the Twitter plugin across Greece and Sweden, led by the IHU and SU respectively. The plugin is addressed to citizens, one of the three stakeholder groups of the project.