During this AMA, Carlos shares his experiences and insights on applying Machine Learning to solve complex testing challenges, his transition to leading AI initiatives in testing, the future of AI in testing and much more!
Task Steps
Watch the āAsk Me Anything on Artificial Intelligence in Testingā with Carlos. You can choose to watch the whole thing (highly recommend!) or choose questions of interest by using the chapters icon on the player or by clicking the chapters from the playbar as indicated with small dots. Take notes as you go.
After watching, reflect on the session and share the takeaway that had the biggest impact for you by replying to this topic. For example, this could be a new understanding of AIās potential in testing or any ethical considerations that stood out to you.
Why Take Part
Deepen Your AI Knowledge: Carlosās experiences and the many topics covered in the AMA provide a great source of information to quickly increase your understanding of thw vast role AI can play in testing.
Engage with Your Peers: Post your key insights from the AMA and see what others think. This is a great way to get different views.
Free Access: Hereās an extra incentive to watch! The AMA recording, previously exclusive pro content, is now freely available to all members throughout March 24. Seize this chance to watch this valuable content for free.
I just finished watching the entire video, and it was very enlightening, great questions and great answers.
What caught my attention was the importance of the context. Using ChatGPT sometimes I forget that its context is the entire internet, or the data you give to it, so the answer is not exactly what you want or expected, and thinking about it to totally makes sense that the future would be the tools that specialize in the context of testing or each language you are working with.
The other topic it got me thinking about is the security of the data we share with those AI tools, we need to be mindful of what I am sharing with it and share only what I can and think if it is really necessary to share it to get the answer I want! At the end, I need to always do better questions.
The entire video is gold, so I highly recommend everybody to watch.
After watching the entire video, I have felt the extreme help that a junior tester can get from AI. Like @elsnoman mentioned at such expertise level, we might not know exactly how to start on something, how to prioritize, risk areas focus etc. And where there are limited resources also, AI would be best buddy to get more insights/feedback.
Also the risk associated with AI not getting trained with the recent data has been clearly explained with shocking real estate scenario happened. The necessity of human in the loop is definitely needed to guard the quality/trust of the AI.
As a whole, it is such a informative AMA session, thanks for sharing!
One of the key takeaways for me was Carloās insights into the future of AI in testing. The discussion on how machine learning can be applied to tackle complex testing challenges really resonated with me. Itās fascinating to see how AI can enhance testing efficiency and accuracy, open up new possibilities for improving software quality.
Additionally, his emphasis on ethical considerations in AI testing to be particularly thought-provoking. As AI continue to evolve in the testing landscape, itās crucial to prioritize ethical practices to ensure responsible and fair testing methodologies.
I completed todayās task of watching the AMA on AI in Testing with @elsnoman. Itās a wonderful resource for software testers and opens up a wide range of possibilities and wisdom for us all.
Here are my key takeaways from this session:
Most people are limiting themselves. Be open to possibilities
AI will replace the Fake Testers / Test Case Checkers soon.
AI has made being good at āhuman aspectsā and creativity back in demand.
Communication
Persuasion
Questioning
Story Telling
Critical Thinking, etc.
Niche Testing Skills
Usability
Accessibility
UX
Observability is a key skill if you are interested in improving your own AI models/workflows. You will need it to boost your confidence, as well as fine-tune your AI applications.
Making your own Model is possible. There is a lot of Open Source material out there. Reference: HuggingFace.co
Specialized AI Tools for Testing will be the things to watch out for. Ex: Postbot.
Companies and testers testing AI need to also ensure that what kind of data should be masked by AI even if itās input into an AI bot by the User.
Knowing the legal and software laws is important in the road to the future.
I think the AMA was great and lift a very good point that you have to think what data you train the AI with that will be the base lline in AI for all decision it makes.
You need to understand what the purpose of the AI is to train it right and also all the time observ what is happening in the world to adopt and change.
But the opertuities are really great if used in right context and for right things.
I think we all in the field need to be looking into this and understand the impact it will make for all of us.
Regarding ethical considerations, since it was explicitly mentioned in the task description, I think this AMA touches the surface only very briefly. Iām very interested in this aspect of AI, & I hope it is the focus of some later challenges
Before watching this video, I hadnāt thought about how using an AI coding assistant in combination with a TDD approach might lead to better results, but - of course! Yes! Seems like a powerful argument to bring to developers who may be resistant to writing tests.
I mean, if youāre using an assistant youāre probably already describing your high-level intention or criteria to generate code suggestions, and writing a failing or stub test would be another (or additional) way to approach that same flow, with the added benefit of having a test at the end, too.
The advice to ālook at the problems you are facing and see which of them AI could help withā sounds very similar to the advice we should give to testers wishing to get into automation. Itās not about learning a specific language, or picking up a specific tool, but about thinking creatively about how you can use the tools at your disposal to make certain things easier/faster in order to have more time for the interesting stuff! Iām here for it!
I have watched the entire video, and I really liked Carlosās approach and since he sees the future of testing with respect to AI, I choose to believe his vision of the future for software testing:
" I think thatās where software testing is gonna be, is I think youāre gonna use AI to do a lot of the automation I think it does that very well. But for everything that automation isnt very good at, well the AI may not be very good at that either.
*An so having humans in the loop to actually analize these things assess them and test them, I see software testers as I think were gonna have a nice boom. I think thereĀ“s software testing is gonna be a pretty awesome skill set to have. *
And I think developers are gonna start moving into that space a lot more as well. Because theyāre not gonna need to have to write so much code anymore, theyll be able to describe what they want and just verify and assess the code that`s given to them and then hook it
"
Thank you for sharing the video. There are a lot of insightful comments in the AMA.
My biggest takeaways after watching:
Be creative
Keep exploring
Some more reflections:
More important to focus on the human element as a software tester to keep yourself valuable in the field, e.g. ability to analyse and assess quality and risks.
In the current time, there are still a lot of good AI tools uprising, keep exploring, donāt just stick to prompt engineering in ChatGPT and missing out the bigger world in testing (havenāt even tried Postbot).
Open source communities like huggingface are gold mines. The video didnāt mention the time and effort needed to gain enough knowledge to be able to flex on using or even creating different AI resources, but it still sounds worthy to invest.
Think about how to solve the context problem when using the tools. As a fan of context-driven testing, hmm, I think even human (or me) cannot always do thatā¦lol
Watched the Ask Me Anything. Here are my thoughts:
Interesting that he would refer to āAIā as Machine Learning which is more applicable.
We still play a huge role in testing with AI. For example providing open source models with extra information on which to make decisions. This is useful for fine tuning models.
Donāt consider Chat GPT etc as a testing tool, but as an aid. It will lack context on the type of work you need to do.
Use AI to help solve issues - debugging code for example. You could have the answer straightaway, taking away a huge chunk of time.
RE confidentiality of data. Quite a big statement that we either take the creators at their word around privacy etc, or you need to create your own. Feel like this is an area that will require some growth as this doesnāt feel sustainable for that long.
Role of a tester is intriguing - might even steer away from the test engineering direction that testing has been going in (whilst remaining important there are more tools that can close the existing skill gap). More of a pivot back towards those āoriginalā testers that excel in the human side of testing - analysis of tests/outputs etc.
A very interesting talk and great to know that there are people out there thinking about testing in AI and how to grow. 99% of the info out there is to do with AI & development so this is encouraging.
watched the whole thing - always good to listen to people passionate. I think a lot here is in not mistaking AI platforms as testing tools - but then, also, the whole thing - AI will not replace testers (or Devs) - that testers (or Devs) with weak AI skills will be replaced by those with stronger ones
How can you assess the confidence your users have in your AI powered software?
Observability techniques are being used. Many AI chatbots have icons where you can tell if you liked a response or not. Sometimes When you click you did not, chatbot can respond with 2 or 3 alternative answers and ask if any of these are better than the one initially presented. The company records input, output, user reaction and chosen alternative output, and uses that to further train the model.
How can we make AI unlearn the concepts learned?
The implicit answer is that you canāt. Instead, you can start with smaller and more specific model, instead of something very generic like ChatGPT. Thereās also a way to tell bot to look into some specific dataset before trying to come up with generic answer. So if you want bot to help with Selenium, you could give it all the Selenium documentation and instruct it to look there for answers first.
What is one use where AI can immensely help a junior tester to test better?
Juniors can get overwhelmed by all the different terms. So they can use chatbot to paste a screenshot, HTML snippet or the like and ask āhow would you <some term, i.e. do performance testing>?ā. Chatbot will come back with some answer, and then you can ask āwhat does that thing mean?ā, āwhy would you do this other thing?ā etc. So it can give more direction.
Another thing is getting unblocked. If you donāt know what exactly to do, you can ask chatbot for some initial ideas, how much time should be spent or some thing etc.
What testing tasks lend themselves better to being done with AI tools?
The generic tools - like chatGPT - are not very good at testing and he does not recommend using them. Instead, you want a tool that has more context about your specific situation. These contextualized and specific tools can come up with scenarios, create tests, write some specific code.
Watch the āAsk Me Anything on Artificial Intelligence in Testingā with Carlos. You can choose to watch the whole thing (highly recommend!) or choose questions of interest by using the chapters icon on the player or by clicking the chapters from the playbar as indicated with small dots. Take notes as you go.
Can you test for biases in AI? You can test for sentiment analysis, is it positive or negative. Use invariance testing and replace tokens within a positive or negative word and see how the model reacts.
Assessing confidence your users have in your AI powered software - usually a prompt has a few answers, some tools as if one response to a prompt was better than another.
What tool are you using for AI testing? Using Langsmith to test Langchaim, or make it observable. Otherwise, use pytest as a test runner, not as complicated as you might think.
How can we make AI unlearn the concepts learned? With open source models you can see what they were trained on, with GPT you donāt know for sure. You can also give GPT files to use as a knowledge base, which is can use first rather than just going out to the internet. Main hub for OS is huggingface.io.
How can we use AI with day to day testing? At the moment its cool but not that useful as it doesnāt have the context. Asking for an API test from ChatGPT will do its best, but if its within Postman and can see your other requests, responses etc it will be much better as it has more context.
How to get into AI testing? Start using ChatGPT to try and solve some of your problems. Donāt underestimate how powerful the tools are. Research prompt engineering.
Security and confidentiality of your data being processed by the AI. If you donāt want to trust a vendor with the data, donāt send it.
Whats the difference between machine learning and AI? ML is algorithims and pattern matching, with AI rather than matching and mapping, it is inferring from data to answer a question.
Where do you see the role of software tester in 10 years? The analytical part of the testers role will still be very valuable, AI does well with structured endeavours with rules and constraints like writing code but less so with analytical techniques. AI would be better at automation for example.
Using copilot to support automation and testing. Using copilot to assist with TDD, write your tests and use copilot to generate functions, classes. Report portal (reportportal.io) to look at test results and check for failure patterns. Starcoder as an open source alternative to copilot.
How can AI help with usability and accessibility? AI can help out with standards (like those in Lighthouse), however an AI doesnāt have to work with keyboard and mouse for example so you canāt trust it.
Where can AI help a junior tester test better? Learning about all the terms that get thrown around in testing that you are supposed to know, a GPT could help you to get started on a testing problem. You can ask other testers but you donāt always have other testers around.
Guard the quality of AI that changes how it behaves in Production. This is known as data drift. If conditions changes after initial model training, it will only ever have the context of the data it was initially trained for.
What testing tasks lend themselves best to being done with AI tools? AI is good at structured work, creating scenarios, writing code. We need to use tools that have the context of what we are working on, rather than considering ChatGPT and Gemini as tools
After watching, reflect on the session and share the takeaway that had the biggest impact for you by replying to this topic. For example, this could be a new understanding of AIās potential in testing or any ethical considerations that stood out to you.
I think I really liked the emphasis on Generative AIās in their web containers are not testing tools per se, they are interesting but not that useful. To be truly useful, it needs to be in context (that word again), like the example of generating tests within Postman with PostBot, rather than just from the ChatGPT response window.
Well, this answers one of the queries I had from yesterdayās task, where I was questioning the practical applications of AI. Great session. My takeaways are:
Iāve built a couple of small AIās using Pytorch (Python AI library). Knowing that I can use Pytest has meant that I can take a look at building unit test code for the AI I have built.
Further confirmation that taking AIās and adapting them is important. I have looked at HuggingFace previously, but I now have a better view of what you can use that repository for.
Great to hear that developers will need to spend even more time testing! After many years of experience with people viewing testing as a ālesserā skill, can we now start ribbing developers?
I will be taking a look at Copilot and Report Portalā¦
Hi fellow testers, this was such a great AMA, thanks for unlocking it. The main takeaways from me were:
Testing for biases - this was really interesting, itās good to know that there are ways to test for an inherent bias but I do wonder how easy it is to even know how to start here, especially if you donāt know what sort of AI model you are testing or what it was trained on
Testing within your context - I think this was a crucial point. One of the issues I have currently with trying to use AI to help me in my day to day testing is that it doesnāt know my context, so I either have to try to give it my context up front which requires me to do a good job in explaining it or continue to narrow down its vagueness into something concrete that I can use. The only tool Iāve experienced so far that knows part of my context is the AI tool within Postman, which looks amazing but in my experience it wrote tests that always passed, even when they shouldnāt, so that was interesting in itself.
Hi! Iāve just finished the video and Iām still in shock.
This is my list:
How important is the context you give to the AI, if you donāt provider the correct one it could be a disaster. The example he exposed about the real state company itās impressive.
How AI can help us to grow our knowledge, I never imagine that.
Motorization is really important to be aware always of the context and there is not a change inside it can destroy your company for wrong decisions.
Laws about data, we need to be an expert or have an expert in the company for this to not have problems with anyone. Maybe, Can we use AI for this too?
Iāve taken note about some tools he mentions, because Iām really interesting in the part how AI can help you doing queries, I would like to know if you give the correct context if you can get the best queries in terms of performance. (This is a big problem in my actual company)
That last question about how the use of AI/ML could potentially lead to making minorities/women/disadvantaged folk less visible and Carlosās answer was really thought-provoking.
His choice of an example where it became very easy to train a resume-screening bot to focus on white males (simply because that happened to be the dominant demographic at that workplace) was an interesting example of the need to remember that correlation is not causation - that is, when youāre looking for people who are say, highly technically skilled and possess certain personality traits, you canāt use names or demographics because the skills and personality traits arenāt bound to those names and demographics.
So, as he said about the data being key, even though Carlos didnāt explicitly say it, we do need to make sure that the training data is relevant and doesnāt accidentally induce bias by way of accidental correlation that has no causative impact on the desired results.
Itās like, if you fed your AI tool a diet of popular female singers because they happened to be the singers that you have on hand, and asked for āsingers like theseā you wouldnāt get any of the male singers you might have enjoyed. (Super simplified, but yeah, if the popularity of the genre is what youāre looking for and the gender of the singer isnāt relevant, you have to keep the gender of the singer out of the data or balance your data set)
I do agree with others that this talk is a gem from start to finish, itās simply that that was the point which stuck with me.
can get from AI. Like 
happened. The necessity of human in the loop is definitely needed to guard the quality/trust of the AI.
Iām here for it! 
