education

Why ChatGPT has Educators Concerned

Michael de RaadtLeave a comment

(I didn’t ask ChatGPT to write this for me.)

Generative AI has educators worried. I am assuming you have heard enough to know what AI-based tools like ChatGPT can offer and, like me, are thinking about the impact on learning. After a disappointing webinar from Microsoft, where we were effectively told to get used to the idea, I’ve been considering what was unsatisfactory about that suggestion and why educators are concerned about generative AI, which brought to mind Bloom’s Taxonomy.

The following diagram represents Bloom’s taxonomy (1956 version) with a few extras. I showed Bloom’s Taxonomy to a gathering of school IT officers and none recognised it. But I’m certain teachers would as they are trained using this theoretical framework. There are multiple versions, but they all represent levels of understanding that can be measured through assessment.

Bloom’s Taxonomy against technological aids

ChatGPT, and tools like it, have been compared to past technologies that challenged educators at their inception. Calculators were, at first, resisted in schools as they could answer mathematical questions that students needed to answer. The Internet, with its search engines and sites like Wikipedia, has also challenged the way students are assessed.

Generative AI has concerned educators to the same degree as these earlier technologies, if not more. Considering Bloom’s Taxonomy gives insight into what may be going on in the back of teachers’ minds.

  • A quick web search can substitute a student’s recall of knowledge.
  • Sites like Wikipedia can provide information showing a level of comprehension.
  • A calculator allows a student to take solve mathematical equation without demonstrating the application of their understanding.

Generative AI, in comparison, can:

  • take a sophisticated prompt,
  • gather information that demonstrates knowledge and comprehension,
  • contrast views in a way that shows application and analysis, and
  • synthesise these ideas into a cohesive written piece.

As suggested in the diagram above, artificial intelligence can now be used to substitute higher levels of understanding that were previously reserved for the human student. Together with the relatively sudden release of generative AI, I believe this is what has teachers worried.

At this stage, the quality of output from generative AI is not perfect, containing factual errors and bias, but it is improving. It is already at a stage where it could be equated to responses expected from post-graduate students, such as medical and legal students. The output is also relatively unique, so existing plagiarism detection is less effective against it.

Going back to my original question, educators are concerned as ChatGPT threatens their methods of assessing student understanding in a new way, which threatens academic integrity and institutional reputation, but also means they have work to revise their assessment approach.

What can be done?

It seems that generative AI is not going to disappear any time soon, in fact it will probably have deeper impacts on education than we can currently imagine. From what I can see, the possible responses for educators to adapt their assessment to this new paradigm.

Block

Blocking generative AI tools, such as ChatGPT and QuillBot could discourage students from attempting to use these tools. This would be a similar response to when earlier technological tools have been introduced and it has been the response from ACT Education (where I work). However, in the case of ChatGPT, the number of places where it can be accessed, beyond its main website (such as Bing and various chat platforms), is proliferating, so blocking may not be effective.

Control

If generative AI tools are not blocked, controls can be put in place to prevent cheating and, perhaps more importantly, ensure students are able to learn and demonstrate their understanding appropriately.

  • Education
    As with other tools, students need to be equipped to understand the associated ethical standards expected. Plagiarism and other forms of academic misconduct are frowned upon and do not lead to successful learning outcomes for students. There already needs to be information literacy training around content from online sources, to allow students to grow into citizens who can identify trustworthy content, and generative AI output is an extension of this.
  • Task Controls
    In mathematical tasks, we ask students to show their working; in essay tasks, we ask students to note their sources. Similarly, with tasks that could make use of generative AI, students can be asked to give more than the answer to demonstrate how they came to their understanding. Assessment design can be improved to deter (or at least complement) use of generative AI by adding specificity to the task or by asking students to deliver artefacts in multiple modalities (other than written text). Ultimately, the best way to avoid cheating is to make tasks seem achievable by providing clear instruction and appropriately weighting tasks.
  • Technological Controls
    Plagiarism detection seems to have been diminished now that generative AI can synthesise a novel text presentation, with appropriate citations. So what can be done technologically to control the use of generative AI?
    • The makers of OpenAI have released a tool that can help detect AI-generated text, which may be useful, but for now it is hard to tell.
    • It’s possible to ask students to present their work using a shared Word document, Google Doc or OneNote file, which shows a history of how the document was constructed, allowing teachers to scrutinise where content may have been copied and pasted. This is not foolproof, but a useful check for teachers.
    • Quizzes have been shown to allow demonstration of understanding that is as good or better than a written essay. A quiz can elicit responses to various question types, which may be useful to redirect students away from generative AI. Quizzes can also be partially or fully automatically marked, which is always an incentive for time-poor teachers. Adding time constraints and a lock-down browser to a quiz should give confidence for most assessment.
  • Physical Controls
    When authenticity of assessment really counts, it still means asking students to undertake tasks in person, away from computers. That could mean a paper exam or an in-person question and answer session. The utopia of online exams, temptingly closes after Covid19 remote learning, will be challenged by generative AI when institutional reputation is at stake.

Embrace

Educators have the opportunity to employ generative AI as a tool for learning and assessment. Like plagiarism detection (which began as a tool for educators to spot cheating, but became a tool shared with students to learn appropriate citation) generative AI in the hands of students can have learning benefits. The possibilities are already being enumerated and I anticipate we’ll see many pedagogical applications of generative AI over the coming years. Here are some.

  • Providing summaries of written works to gain a better understanding
  • Generating exercises to self-assess understanding
  • Supporting text revision where non-native language skills are lacking
  • Providing a starting point for in-depth research

Evaluation is a level of Bloom’s Taxonomy that I don’t think AI has yet conquered and that leaves room for higher-order thinking to be assessed. A colleague pointed out an article from the Sydney Morning Herald describing tasks being prescribed to medical students, who were instructed to prompt ChatGPT for information and then critique it.

Conclusions

The benefits of generative AI could lead to better student outcomes, if educators allow learning to make use of them. At present, there is significant positive innovation to match the reactions of those wishing to block generative AI.

I don’t expect efforts to block generative AI to last long, especially while they are less than fully effective. Ultimately a point of balance between control and embrace needs to be established, where assessment of understanding can happen and the learning benefits of AI-based tools can be achieved.

Limitations

I need to say that these views are my own and not an official policy or stance of ACT Education.

I haven’t completed a comprehensive survey of opinions from “the coalface”, however I’ve been communicating with people who have been gathering reactions to generative AI and writing briefs and presentations for Education decision-makers, which led me to this understanding.

An EdTech Project Management Process for Schools

Michael de Raadt2 Comments

When I started as a leader responsible for Educational Technology projects in a school, I lacked a framework to work within. Having come from the software development world, I understood how to develop systems, however, selecting and implementing systems is quite different.

I looked into a few frameworks for system change management and found PMBOK (the Project Management Body of Knowledge) was adaptable to EdTech projects in schools. PMBOK is a general project framework, but it is adaptable, so I set about writing a process guide and template based on it and also included a number of software and educational models to give specific relevance to schools.

In 2018 I ran a workshop at the AIS ICT Leadership conference to share my version of this process and it was well received. In summary, the process involves a number of work areas for planning, execution and change control as shown in the following diagram.

When working with a project team, a template can be used as the basis of a collaborative document to work through the planning work areas.

The most involved area is the Procurement area, which involves consultation to determine requirements, transforming these into prioritised criteria then setting a cut-off for essential criteria.

The documents below describe the process in detail including a comprehensive guide, a project plan template and the slides from my workshop (with examples of a software and hardware project in a school).

I’ve since heard back from other schools who have applied the process successfully.

I’m sharing it here now so that I can refer back to this process as I describe an LMS transition project we have undertaken over the last two years in subsequent posts.

Organising Selection for an IT Position

Michael de Raadt1 Comment

Over my career I’ve been involved in interviewing and selecting new staff for IT positions on numerous occasions. I’ve learned a few tricks along the way and I thought I should share those.  A lot of these techniques are generalisable to positions peripheral to IT and elsewhere.

Choosing a candidate

The Panel

It helps to have more that one person doing the interviewing; two is OK; three is ideal; four can be intimidating. If the position is really serious and more people need to be involved, create two panels with different foci.

Interview

Aim for diverse perspectives in the panel members. If you’re a manager, involve a technical staff member and a support staff member, such as someone from HR.

Preparing

  • Short-listing
    Without going into too much detail, the panel that will interview should be the ones selecting who should be interviewed. Start separately and blindly review all the CVs. Bring together opinions into a collaborative space, such as a shared spreadsheet.
  • Discuss candidates openly
    After each panel member has rated candidates, come together to decide who to interview. Be open to disagreement as others may have spotted potential that you have not seen. Consider rounds of interviews with the most likely candidates first.
  • Invitations
    When you have a list of candidates, you need to invite them in.

    • Negotiating a time is best achieved over the phone. Offer the candidate opportunities within a specific window, but be accommodating.
    • Once a time is set, send a formal invitation that introduces the panel and their positions; this establishes perspectives for the candidate. Set expectations for where to go, when to arrive, what to wear and how long the interview will take. You may want to prompt the candidate to undertake some research into your organisation by directing them to online resources and work spaces.
  • The script
    It’s good to have a set series of questions going into the interview. All panel members should agree on the script before interviews start. Use a common document with names beside each question (rather than each panel member having their own script); this allows you to pass the flow of questions between panel members. If you have a script from a previous position, review the questions and ensure they are relevant to the current position. The script can be duplicated for each candidate so that notes can be inserted, by someone not asking current questions, during the interview.Interview script
  • Quick recap before interview
    Before an interview, all panel members should take a few minutes to review the candidates CV. Discuss their strengths and peculiarities so that you can focus questions during the interview.
  • Hospitality
    In a sense, your organisation is being interviewed as well as the candidate. You want the best candidate, who could possibly go elsewhere, to choose you. Simple things will help, like:

    • tidying the space where the interview will take place,
    • ensuring the temperature is comfortable and
    • having glasses and water poured for the candidate and the panel.
  • Everyone shakes hands
    Allow the opportunity for each member of the panel to shake hands with the candidate. That first physical contact is disarming and will establish what could be a future working relationship.
  • Seating
    Don’t arrange seating in a way that is confrontational, such as sitting on the opposite side of a big table from the candidate; a small table is better, with the candidate as part of a formation that is inclusive, like a circle.

The Interview

Repeat Introductions

Start by reintroducing the panel and what they do. This can be quick, but is important to preface the questions you will ask later. The panel leader can do this or each panel member can quickly say who they are and what their role is.

Questions

The candidate’s CV will tell you about their education, their experience and their skills, but it won’t tell you what kind of person they are, how well they will work with you and how they can apply what their skills. You want a good script of questions that tease these important aspects from the candidate’s brain.

brain-2029391_960_720

  • Icebreakers
    Candidates will be mentally prepared to convince you about their professional worth, but don’t jump straight into serious questions. Start by allowing the candidate to settle in and feel comfortable. A good way to achieve this is to ask the candidate to talk about their personal life; if they start drifting into work and skills, redirect them by saying you will get to that soon.

    • How did you come to be here in ___ ?
    • Tell us a bit about yourself as a person. What do you do in your spare time?
    • Tell us about your study. What inspired you to get into IT?
  • Focused career questions
    You want the candidate to tell you about their experience, but you don’t want a litany that will take up all your interview time. Ask questions that will allow the candidate to showcase them self, while highlighting aspects you are keen to hear about.

    • Without going into too much detail, tell us the places you have worked and your roles there.
    • (If applicable) Why are you leaving your current position?
    • What has motivated you to choose your career path?
    • What are some of the tasks you really enjoy doing?
  • Tell questions
    It’s hard to tell when people are being honest. One technique for eliciting humility and honesty is to ask the candidate to admit where they have failed. This may be counter to what the candidate is prepared for and it may be affected by cultural background, but it can give you a good idea of whether you want to work with that person. It’s a good way to distinguish potential assholes.

    • Can you think of a time when things did NOT work out the way you expected them to?
    • Can you tell me about a time you had a conflict with a colleague? How did you deal with it?
  • Focused skill questions
    You should be able to tell what skills a candidate has from their CV, but you want to know if they have real experience or was it something they observed someone else doing.

    • Tell us about your experience with Active Directory?
    • Have you ever written documentation in a wiki? No: what did you use?
    • Have you ever worked with a issue tracking system? How was that used?
  • Don’t forget the soft skills
    It’s easy to get stuck on technical skills for an IT job, but non-technical skills are really just as important in the day-to-day working of a successful team.

    • Have you worked as part of a team? What was your role?
    • What techniques do you use to manage your time?
    • How do you handle conflicting priorities?
  • A conundrum
    You want someone who can ‘think on their feet’ and consider alternative solutions. Posing a scenario that seems unsolvable at face value will prompt candidates to demonstrate their ability to think ‘outside the square’. The following example is for a service desk position in a school.

It’s been a busy day; you are feeling under pressure and a teacher calls you demanding that you set up an email account for a person who is not an employee but has come into their class to present.  This would be against the school’s policy, but you understand the teacher needs to make the class work. How would you deal with this situation?

  • Most candidates will start by stating that they cannot break policy because they want to give you the impression they are honest workers, ready to follow the rules. Some might say they will seek permission from a manager to break the rules. A good candidate will recognise that problems are often not what they are first reported to be and probing into the client’s needs will allow you to consider the problem then create a solution or a workaround.500px-Tribar.svg
  • Questions about your organisation
    You want to know if the candidate is actually interested and enthusiastic about working in your organisation. Give them the opportunity to share their research and how they have envisaged them self in your organisation.

    • What do you know about ?
    • What do you think it will be like working in a ?
  • Prompt for their questions
    Allowing candidates to ask you questions is more than a courtesy, it allows the candidate to take control of the interview and demonstrate their strengths and knowledge by probing you about what you do, what technologies you use and how the organisation works. A good candidate will come with prepared questions.

    • Do you have any questions for us?
  • Formalities
    Don’t leave yourself open to surprises.

    • What are your obligations and availability?

If you are leading the panel, avoid keeping all the curly questions to yourself. Farming some complex questions to another panel member allows you focus on how the candidate is answering the question, following their body language and ‘reading between the lines’.

Flow

A smooth interview is not rushed, nor is it slow. With good flow, the interview can be comfortable and friendly and elicit the honest answers you are seeking.

comments-151907_960_720

  • Build up with some easy questions first.
  • Hand over between panel members when asking questions.
  • Ask questions from the panel member who has the perspective from which you want questions answered (personal from the manager, technical from the technician, organisational from HR).
  • Be adaptive.
    • Don’t stick to the script when you want to clarify or probe deeper.
    • Don’t ask questions that have already been answered.
    • Make questions specific on-the-fly.

After asking all of your questions, lead into a task…

Evidence

A candidate may say they have the skills you require, but it’s hard to judge to what degree that is true. Their CV may have been developed over time, with outside help. Every candidate will say they have good communication and problem solving skills; we all have a self-optimistic bias. Don’t be afraid to take some time to get the candidate to demonstrate their skills.

cat-1800843_960_720

  • A role-play
    Pretend to be a client with a predetermined problem. Ask the candidate to put them self into a support role and attempt to unravel the problem. Getting the answer is not as important as how they approach the problem.
  • A quick quiz
    Allow the candidate to answer questions in a quick quiz. You might throw together some basic questions in a Google form or online survey and ask them to provide their answers.
  • A writing task
    Being able to write clearly is an important skill for all IT workers. Set up a scenario and ask the candidate to respond to a pretend client. Writing a pretend email or ticket-update on a machine you provide is an easy way to run this task.
  • A dev task
    If the candidate is applying for a technical role, ask them to resolve a bug or simple problem. This may be something they have to do after they leave you and later submit the response back to you. Be sure the problem requires them to establish a dev environment close to your work environment.
  • A presentation
    If the candidate is applying for a role that involves training, ask them to run a quick training session on a simple technology. If you’re considering this task, you will need to give the candidate notice before the interview so they can realistically prepare.

Debrief

Assessment biases can creep in over time. You can glorify earlier candidates or favour candidates you have seen more recently. Reflecting immediately after each interview is recommended, even if this means delaying the next interview by a few minutes.

When you’ve seen all candidates, hopefully you’re in a good position to choose. If none of the candidates are suitable, consider re-advertising. If there is a candidate that is suitable, but you’re not completely confident, remember that you can rely on a probation period if things don’t work out.

Don’t forget the unsuccessful candidates. Failing to respond respectfully to unsuccessful candidates puts the reputation of your organisation in danger, whereas an honest response with feedback that will help the candidate in future will be welcomed.