Alex Kolokolov, a data expert and bestselling author of “Data Visualization with Microsoft Power BI,” the founder of Data2Speak Inc, along with his team, actively trains students and professionals in AI skills. They conduct courses and week-long marathons, regularly performing crash tests and studying the capabilities of artificial intlligence.

AI is rapidly replacing employees in automatable tasks. At Data2Speak, we tested AI’s ability to handle a data analyst’s daily work, tackling challenges in data processing, modeling, metric analysis, visualization, and reporting.

Initial task for a data analyst sounds like this:

“Analyze the table of webinars and courses and create a text report for management on conversion rates, success, and popularity of webinars and instructors. Calculate the ROI of each webinar.”

Model Creation and Data Processing

The CRM export with webinar data consists of two sheets within a single Excel workbook: one sheet contains data on promo webinars, including topics, authors, audience size, and the cost of attracting attendees. The other sheet records course sales among people who attended the webinars.

As shown on the image below, the data can be easily matched using the Webinar ID field.

Sheet 1 “Webinars Data” + Sheet 2 “Course Details”:

*All names in this article have been changed to maintain confidentiality and protect financial information. Any resemblance to real individuals is purely coincidental.

First Challenges Encountered

• Microsoft Copilot refused to work with the .xlsx file—the most common format for Excel documents. Instead, it suggested using .pdf and .png, which are completely unacceptable formats for final reports.
• Claude.ai does not support .xlsx files but handles CSV files well, though it struggles with analyzing two tables simultaneously.
• DeepSeek inconsistently read the data, possibly due to temporary issues.
• GPT-4 handled the upload perfectly, so we decided to proceed with it.

Next, we followed a simple method—asking the AI questions as if it were a regular analyst on the team.

“Please determine:

• which webinar subject converts better to a course purchase
• the best performing hosts, according to the conversion rate
• the most popular subjects
• ROI of each webinar”

The AI performed well in answering questions but struggled with accuracy in ROI and conversion rate calculations. The errors arose from its need to repeatedly reference and merge datasets.

Upon reaching the free version’s data analysis limits, we upgraded to GPT-4o—turns out even an AI intern requires a budget!

While it can’t build data models, it links tables effectively. However, its output contained multiple rows per webinar due to purchases occurring on different days, leading to calculation issues.

We refined our prompt and achieved the desired result: 1 webinar = 1 row. This step highlighted AI’s limitations in handling tabular data. Here are the key issues we identified:

• 1 task = 1 table: AI models may lose context and become confused when handling larger datasets, such as multiple sheets within a single workbook.
• Internal formulas = a mystery: If tables contain logical relationships, AI does not interpret them correctly.
• The quality of your prompt determines the quality of the result.

⚠️ Warning! Always verify AI-generated results as you would with a human assistant—AI can be just as inattentive!

Key Metrics Calculation

Now we are ready to fully work with the data. We asked the AI to suggest relevant metrics, selected the best options, and attempted to calculate them. Here is the final list of key metrics:

• Webinar conversion rate: Identifying which webinars most effectively convert attendees into course buyers.
• Top instructors by conversion: Determining which instructors have the highest conversion rate from webinars to course purchases.
• Most popular webinar topics: Identifying the most attended topics without linking to specific instructors.
• ROI per webinar: Calculating the return on investment for each webinar, independent of instructors.

Each step requires verification since AI has a short memory—by the end of a conversation, it may forget earlier instructions and recommendations.

For the first three metrics, the AI provided perfect responses. However, when calculating ROI per webinar, the AI generated a list of 50 rows instead of the expected 10 unique webinars.

We highlighted this mistake to the AI. At first, it did not immediately acknowledge the error. Only after addressing the issue did the AI provide the correct result.

The Insight: Trust, but Verify!

Test complex queries on small data samples first to identify logical weaknesses in calculations. Keep reference values for comparison—column totals, unique record counts—using a reliable BI tool (even Excel).

Summarization and Data Visualization

Now, we add visualizations to make the results easier to interpret. The AI handled this without major errors (see the picture on the left). However, for those who are meticulous or familiar with “Data Visualization with Microsoft Power BI,” some obvious refinements were needed: adding data labels and removing gridlines and axes, which were unnecessary in this case.

Before and after:

Working with textual data in AI is easier, which often results in more interesting responses. However, AI can sometimes generate information beyond the given data. For instance, when analyzing the chart for the best-converting webinar, it decided to provide generic marketing recommendations instead of basing its conclusion on actual data.

This is a common issue—it’s important to specify that we need not just general insights and recommendations but concrete tasks based on our real data. When formulated this way, the AI’s response becomes much more valuable.

Conclusion: It’s great that AI can not only create but also improve charts, but without human oversight, they still end up being difficult to read.

Plan ahead which key information each chart should convey, and check it for visual clarity. The book “Data Visualization with Microsoft Power BI” can help with this.

Crash Test Results

It’s still too early to replace human analysts—AI is more of a timid intern than a full-fledged expert.

Key Takeaways from This AI Experiment:

• AI does not support all file formats (each model has its preferred ones).
• Constant verification is necessary—AI confuses numerical dependencies.
• Multiple iterations, refinements, and corrections are required. No automation.
• Difficulty in generating structured reports—manual intervention is still needed.
• Inaccuracy increases with data volume.

Success with AI in data analytics requires clear task definitions and ongoing oversight. While AI excels in calculations and visualizations, it needs precise instructions. Critical thinking remains essential—AI can make mistakes, so always double-check information. As long as you think critically, machines won’t replace you.

Author: Alexey Kolokolov, data analysis expert, bestselling author of “Data Visulization with Microsoft Power BI,” and founder of Data2Speak Inc. He trains students and professionals in AI applications, organizes courses and intensive programs, and regularly tests neural network technologies to assess their effectiveness and potential in data analytics.