Five analytical tasks in under a minute
Less than a year ago, one of my AI experiments went viral. I decided to see how far I could get in marketing a product launch in under 30 minutes using the then-new Microsoft Bing and ChatGPT. The results were impressive. In half an hour, I had a draft email marketing campaign, social posts, and even a landing page and video. To me, it was an early sign of the potential of AI to do real work.
In the 10 months since, the evidence for AI as a productivity booster has only grown, but many people are still not even trying to use AI (though that isn’t true for all populations - almost 100% of my students this semester reported using LLMs). Maybe it was the fact that it took a little skill to use LLMs well. Maybe it was that people found using AI unnerving and gave up on it quickly. Maybe it was the fact that organizations frowned on AI use. It certainly didn’t help that the ubiquitous chatbot approach to AI hid a lot of its power, which was only revealed after hours of experimentation. As a result, AI use was mostly something a small proportion of people in organizations used, often keeping their applications secret to get the maximum value with minimum risk. The extent to which AI was truly disruptive was hidden.
I think that changed this month, with the wide release of two tools that made AI use much easier and, soon, harder to ignore. Microsoft’s Copilot for Office became widely available for anyone for $20 a month, and OpenAI’s GPTs became more useful, with a combination of a new “GPT Store” and team-based subscriptions. Neither of these tools fundamentally changed anything about the capabilities of AI. They both use GPT-4 which is well over a year old, though it is still the best AI model available, at least for now. Instead, these systems made AI use feel much easier and more normal, removing the uncertainty (and some of the power) associated with directly using GPT-4. Now, companies are buying tens of thousands of Copilot licenses and the most popular GPTs are being used hundreds of thousands of times. AI use is becoming normalized.
And I think many people are not ready for what happens next.
To see why these tools can be so disruptive, I want to revisit my old experiment of giving myself 30 minutes to launch a product. I last did this 10 months ago — forever in AI time — so it seemed unfair to give myself a half hour again. Let’s try doing the work in under a minute.
I also decided to up the difficulty level in another way. I would do five different tasks, rather than one: I would launch a product and write a market research report and create on-trend designs for a kitchen and make an entire PowerPoint and craft a syllabus. All at the same time. All in 59 seconds.
My screen showing the five windows before starting
With my five windows set up, it was time to start. I did some minor testing of prompts to see if they worked before this experiment, but I did not attempt to optimize any of the prompts or outputs, and used the system in the way I see first-time AI users do. Here is what I set up:
- Microsoft PowerPoint: I used the default Copilot option to turn a file (here an AI-written business case about Tesla) into a presentation.
- Microsoft Word: I used Copilot with a simple prompt: Write a full syllabus for a 6 session introductory entrepreneurship class including tables, summarize the main class learnings, include assignments and grading.
- ChatGPT with a version of my Trend Analyzer GPT, which has a short prompt that asks the AI to search for trends and then photoshoots of on-trend designs.
- ChatGPT with my ProductLaunch GPT, which basically combined all of the commands that I had used in my previous product launch post, and had the LLM automate that process. I told it to look up a Wharton Interactive product - the Saturn Parable, and it did the work from there. (Neither of these GPTs are particularly good, I put them together as an experiment - but feel free to try them)
- Bing (or Copilot? Microsoft keeps changing names) with the prompt: write a draft market research study in the style of a top strategy consulting firm on the market for virtual reality and augmented reality devices, use market research and discuss trends. Bing was actually the most problematic of the systems, repeatedly telling me it was unethical to ask it to write a market research report for me, until I called it a “draft market research study” which seemed to satisfy the system that I was acting appropriately.
Then I started the timer and hit enter in each window. You can see the real-time video below.
Five reasonably high-quality drafts were done in under a minute. I had renderings of a few trendy kitchens, a three-page syllabus (that wasn’t half bad), a thirteen-page slide deck with speaker notes, an almost 1,000-word market research summary, and a product launch strategy (with draft emails) for one of Wharton Interactive’s teaching games that was really solid. A few more interactions with the AI, and a bit more time, and they might have been excellent.
My screen 59 seconds later.
If you last checked in on AI a few months ago, you might also be surprised at how much the quality of the output has improved. While there are still errors and hallucinations, they are increasingly subtle and rare, and the quality of even first draft output has improved tremendously. It isn’t as good as good human work, but it is surprisingly capable compared to many writers. They also require much less expertise, as you don’t need to write good prompts to get good outputs, making AI more accessible.
And this accessibility and quality is exactly what is going to cause a crisis.
For many people in many organizations, their measurable output is words - words in emails, in reports, in presentations. We use words as proxy for many things: the number of words is an indicator of effort, the quality of the words is an indicator of intelligence, the degree to which the words are error-free is an indicator of care.
When a middle manager writes a weekly report on the status of a major initiative, the report may not be the point. Instead, it serves as a signal that the middle manager has done their job, speaking to the relevant employees, keeping an eye on the status of the project, and making corrections as needed. And it has always worked well enough - a senior manager could tell at a glance if the report was seemingly substantive (showing effort) and well-written (showing quality). But now every employee with Copilot can produce work that checks all the boxes of a formal report without necessarily representing underlying effort.
What this means is not yet completely clear. In organizations bogged down by meaningless paperwork, it may help as the endless procedures of bureaucracy is taken over by machine created, and filled out, forms. Other processes that would be meaningful when done right, but which are too often done purely to check a box, will suffer more. Performance reviews, for example, will likely lose all of their value as managers everywhere have confided in me that they are using AI to make reviewing easier. But the results could be even more severe as some employees may face a crisis of meaning about the nature of their work when faced with AI written content that replicates their work, but not their thought. What does your skill and effort mean if people don’t care if your work was done by a machine?
And the kicker is that the quality of GPT-4 writing is quite good, and, when given access to a source document or data, hallucinations are quite low. So why not use it? The temptation of the write-it-for-me Button, as I have noted before, is ubiquitous. No one is going to write their own drafts anymore. And very few will seriously edit those drafts either, as our research shows that people “fall asleep at the wheel” when faced with a good-enough AI. AI content will suddenly be everywhere, in every organization.
Thus, to use AI at work requires you to think about what your work means to others, and what it means to you. These are answerable questions for thoughtful organizations. But very few leaders seem to be thinking about these issues as AI adoption expands.
Yet, on the other side of the crisis lies the possibility of freedom. Of the five tasks that the AI did for me in less than a minute, I have done four repeatedly over the years (creating a presentation, launching a product, making a syllabus, and drafting a market research report), but I only really would enjoy doing one today - creating a new syllabus. I am more than happy to delegate at least part of the other tasks to the AI. That is why surveys repeatedly find that workers like to use AI, even while recognizing potential risks to their jobs. The AI does the work they do not want to do. And tools like Copilots and GPTs make it easy for anyone to figure out ways to delegate drudge work, so they can focus on what they actually like to do, and what other people value about their work. Organizations that figure out how to embrace this form of flourishing (and are willing to cut processes and approaches that no longer make sense in a world of AI writing) may find themselves benefitting.
Even more than that, there is the opportunity for expansion of what we can do as flawed and limited human beings. One of the tasks the AI completed, creating trendy interior designs, has always interested me, but is completely beyond my natural abilities. But with AI help, I can start to explore a new set of interests. Beyond freeing us from tedium, there is the fascinating possibility that AI can help us expand our own capabilities. But this isn’t going to happen automatically.
To paraphrase Kratzenberg’s First Law: “AI at work is neither good nor bad; nor is it neutral.” AI does not automatically improve the experience of work, nor does it automatically rob us of meaning or replace workers. How leaders and employees use the technology will determine whether it is good or bad. But AI also isn’t neutral. The use of AI will inevitably lead to deep and profound changes. We shouldn’t pretend those changes aren’t going to happen, and we have to take responsibility for determining ways of using AI that emphasize the good, and not the bad.
Recommend One Useful Thing to the readers of Ulrich’s Newsletter
Trying to understand the implications of AI for work, education, and life. By Prof. Ethan Mollick