From the month of February this year, I was drawn into AI, Claude, and agents. The idea of using new emerging technology like AI chatbots, for personal learning and skill development caught my attention and I was obsessed for couple of months. Lot of time, effort and energy went to these topics, I was clueless for most of the part. The objective of this blog is to condense all of experiences, mistakes, what worked and what did not into single blog.
Now if you noticed the title of the blog, instead of using LLM, I have intentionally used the word “chatbot”. Because modern chatbot uses LLM (large language model) for reasoning and text interpretation and answering, but does include some of the agentic capabilities like, retrieving information from PDF document, doing web search and web research and generating images.
Understanding how elite use chatbots, AI frameworks and agents.
I was fortunate enough to attend Jason Haddix’s training on “Red Blue Purple AI”. And after the training, I spent some more time on YouTube and discord trying to understand how best (people with decades of experience) are using AI. It all boiled down to two things
1. Build AI clone of their own skill sets, and understand how AI can be orchestrated autonomously to get the task done:
In other words, elite professionals build skills, agents sub agents, and harness and orchestration around their trade (skills) which they have built. This is two fold, one is to have real skills and that real skill translates into AI systems. If you ever heard bug bounty hunters or web researchers using AI, they all tell that they have given their old reports to Claude and ask it to analyse. Understanding this is very important because all bug hunters are exceptionally good. They could have given AI the reports of people who are on top of leaderboard. But instead, they choose to fine tune AI skills to their own experience. The second part is to understand AI features like skills, agents, sub agents and how to extend Claude’s ability to orchestrate (run) agents, and how to improve existing harness (giving agents an environment configured with tools, and resources to get the job done). Don’t worry if the topics like harness and orchestrate throw you off the hook. You can watch some youtube videos from IBM technoglogies or read this blog which is by Daniel Miessler on the topic of Real World AI definations.
2. **Use AI to remove friction: **
Security work is often messy, and in order to find critical bugs, we are exposed to repeated work that drains our energy and life. So, from the expertise the professionals have built over the years, they have configured the AI setup, in such a way that AI does that specific thing. Now note that the way elites use AI is not as simple as find critical bugs. They OFTEN fine tune to single test case and mention how to do that task to smallest detail or have build AI skills around that. Note that skills here is in reference to procedural knowledge that we often give to AI agents to get a job done.
Now that we have picture on how the elite use AI, the question I asked myself is, what is there for me? I am new to industry, and I am yet to build expertise, skill sets and knowledge right? Understanding how elite use AI gave me understanding of how to use AI, and how not to use AI. I was surprised when Anthropic advised using AI output to check and verify against real work not the other way around. The 4D Foundation on using AI with was eye opening for me. It’s a free course and my main take away was always
be diligent by becoming capable enough to do the work with or without AI and comparing AI output with real work, and apply discerning to judge what is accurate and what is not
In simpler words, always cautious about AI generated output, and to always verify and check against real human’s work.
So why is this blog about using chatbots and not AI agents?
One of the primary reasons is that latest development with chatbots and AI models are surreal. One test you can do is pick up your best technical work, finding a bug or auditing an application. Explain the technical observations that lead you to find that bug or technical work. Ask AI what can be done here, and what can happen here? I was surprised that AI was able to deduce and guess the bug and technical outcome. Additionally, I have also come to a positive conclusion that premature use and over reliance on agentic abilities so solve manual work without building enough muscle memory, understanding of work is an recipe for disaster. Even if you get the work done, unless you can explain what you have done, and why, you are not making any progress. **If your answer to why you have opted for this approach or this solution is “AI have suggested me” then we really have not built the skills. ** One should be able to answer our choice, and we should merely try to use AI as means to gather information, or to have an opinion. It should never create professional decision that lies on our shoulder.
And also if we dive straight to AI agents, without understanding capabilities and limitations of chatbots, we are limiting our usage and understanding of AI, right? for these two reasons (one is to understand capabilities of AI, and second one is to limit AI usage and reliance during skill building or learning phase), we are making a blog on using chatbots effectively for our learning and skill development.
Intended audience for the blog
This is not a blog that teaches about some niche prompt, that solves our learning problems, or a secret method to improve our learning. At the end of the day, no matter how much information you gather with AI, ask it to make notes for you, we still have to sit and put deliberate effort to understand and learn. The intension for making this blog is that I have struggled so long focusing on wrong things, prompts, skills and gimmicks.
The intention of this blog is to make an honest attempt to steer away from techniques, and to focus on features of chatbots, and to give an idea on how it can fit in our learning goals. Techniques prompt can get outdated, but our deliberate efforts to make best of technology stays.
Lastly this blog is not about me saying the right way to use AI. It’s about making you discover and explore a deliberate and intentional way of using AI chatbots, that works the best your learning goals. It all starts from understanding abilities of AI chatbots from first principles and then trying to fit in our learning goals, not the other way around.
Core foundation principle discussed in this blog.
Now how can we learn core principles without focusing on prompts, techniques and copying other’s AI agent skills and wasting tokens?
This circles back to 4 - D principle that is taught by Anthropic on their course “AI Fluency” which you can watch on YouTube or learn from their academy. For me my personal takeaway, (which we have covered in remove friction with AI section too) was
Be diligent by learning your trade, asking right questions, applying right mental models like first principles thinking, second order thinking, invariant, and then apply your skills experience to discern the output from AI, and use AI to extract and reformat raw knowledge in a way that’s useful for your learning.
Let’s be real, in my observation present models are best when we use AI to gather knowledge and right resources, but we still can’t outsource our thinking and understanding. We still need to sit, think ask questions, struggle, fail, adapt, and become better by constantly trying and improving in our trade.
Again, diligence at work, and learning can be practiced through continuous and sincere effort and attempts made at learning and getting feedback. Without feedback from real and experienced people, mentors, our progress can get stagnated. Of course there are lot of ways to practice diligence, one way to start is through making a real attempt at learning and getting feedback in our understanding through reaching out to people who are one step ahead of us, or through AI. (Reaching out to people is preferred). Never under-estimate the importance and role that feedback does, because there is no other way to know what we are doing is good enough or if it could be improved. Of course, you can figure out all this alone but it will take up lot of time, and mistakes, so it does help to seek feedback, after our full attempt on doing something or learning something new.
Discernment is another important cornerstone. It’s about deciding if the given text, blog, or an output generated by AI is useful enough for our work or not. Anthropic also made an entire course on this one, but this is accessible to Anthropic partners, which you can sign up for if you have work or professional email. My main takeaway was to understand the fact that:
Just because AI drafts an text that appears to be confident, does not mean it’s accurate. We can use our experience and skill to judge the information cited by AI (Also check out pro tip 4# that helps to reduce hallucination and fallacy). Discernment is an real skill that an be built upon, and it all comes down to our ability to judge any text and analyse if it’s accurate or not.
The idea of “Diligence” & “Discernment” is really important, and it does compound in our attempts to learn any new topic or skill. Even if you don’t wish to try any of the tips or connectors, or MCP or skills from this blog or from online, if you take away the ideas behind these two terms, that would be more than enough for me.
What makes a chatbot? what are it’s core features that I should be familiar with?
Chatbot consists of 3 main core features.
1. Projects / / Notebooks. :
These are used to group and categorize chats based on topic. Note that in Claude and ChatGPT it’s called project, and in Gemini, Notebooklm and in copilot m365 it’s called notebook.
2. Skills:
As we have talked previously skills are procedural knowledge on how to get a task done. Best way and the ideal way would be to build skills based on your expertise and knowledge, not from internet.
3. Memory:
Chatbots store important details about you, your goals in memory. You can search around in setting of your chatbot to find this. We will be discussing in detail about all 3 aspects and how to use it for our learning.
How does knowing features of chatbots help me to use it effectively for learning new topics?
Now that we understand features of chatbots. We can start using them. Most people including me never categorize and filter chats into their respective projects. Let’s say I am brainstorming some ideas that I have read from an fiction, and next day I am asking chatbot about how to test DOM XSS manually without “DOM INVADER”, and some time later I ask which mutual index fund is good for getting “X% of consistent returns?”, it does sound messy right?
Based on our use cases, we can and we should create projects. By default on non pro account Claude gives you 5 projects. I don’t think there is some restriction with ChatGPT as of today. Here is how I categorize projects.
This is based on my use case. I use ChatGPT to the following activities:
- To learn about AI agents, how it works and to understand words like harness
- Enhance any pictures I take with family. (Don’t do this if you are not comfortable sharing your pictures to an chatbot for privacy reasons)
- Personal coach to discuss ideas I come across YouTube, Instagram, and to have interesting conversation about logic, reasoning, philosophy etc
- To learn web pen testing
Now, usually if we use start using projects we get following benefits:
- In “Claude” projects act like RAG and extend memory by 10x.
- It’s way more organized and helpful to have discussion with chatbots. Anything that’s one time use either goes outside projects or in temporary (incognito chats).
- Context gets separated, and we give AI an equal opportunity to understand our goals.
- We can set instructions on tone, the output we expect from Claude or ChatGPT, for each of projects. We can also set memory if the memory from project should be shared outside project. (This has to be the coolest feature, and we get to have personalized experience from chatbot, just the way we expected the chatbot to answer as per our preferences). Think of this as setting up system prompt for each use case.
So if you are using chatbot, start today by having an habit of using projects, instructions, and memory features.
Pro tip 1#:
In “Red Blue Purple AI” Jason Haddix uses chatbots to create payloads for fuzzing, to understand topic, and shares his methodology on creating a chatbot with his custom instructions. You can start using projects, the way you wish to use. Sky is the limit. You can start using it for learning, testing, payload generation, wordlist generation or any use case that fits your needs.
Pro tip 2#:
Having access to enterprise Microsoft copilot 365 allows you to add more than just chats. In copilot it’s called notebooks, you can add teams chat, e-mail threads and feed in important professional discussion and chat to the work you are doing. You can also attach any document without violating rules imposed by your organization. Also keep on eye open on “Cowork” as it’s allows you to create your custom skills and use built in MCP (if your organization allows MCP).
Pro tip 3#:
Best way to use chatbots is to have separate project / notebook as per your use case, with custom instructions, that defines how chat bot should answer, and treat you, and with added skills that contain procedural knowledge on how to get your work done. Example of this could be, an separate project for auditing azure resources, with instructions saying how the response and tone of chatbot should be, tied to skills that teach how AI should search the Microsoft documentation for this particular audit control.
Pro tip 4#:
To avoid hallucination, in your instruction or in your skills or in both, mention that, AI should always cite information along with line or page number from a reference document or website, along with confidence score (which could be percentage or a number between 1-5 or 1-10). And when Claude, ChatGPT is unsure, ask it to admit and tell you that it’s unsure. This tip was shared in Claude conference talk. And this completely improved the responses that I was getting previously for my work and learning.
About NotebookLM
This one hour talk gave me lot of new ways to digest information from lot of resources. Let’s say I need to learn about an azure resource, or web attack vector, instead of reading 5-10 blogs, if I am short of time, I can put those references into NotebookLM and ask it to generate slides, infographic, quiz, short 3-7 min video that is animated, and an podcast with voice of two people who talk and discuss about core ideas from those references. It’s too amazing right?
One use case is that I initially struggled to really understand difference between BOLA and BFLA in API testing. Generating visual content (slides) helped me out. Check out the images that are generated by NotebookLM
You can prompt NotebookLM, and there are some online resources on tweaking the slides, images generated by NotebookLM that you can find on Github. The bottom line is if you were visual or audio learner or find something that is hard to understand by text, please do leverage NotebookLM.
Pro tip 5
You can also use ChatGPT models to generate infographics as a way to summarize your learning from a study topic. You can ask copilot with ChatGPT models to generate infographic on a topic you have recently studied with all important points or sub heading given in your prompt.
Understanding memory in chatbot
Memory is text that AI uses to remember something about you, so that it customizes the responses to match your requirement. It’s recommended to check memory, in chat, filter and remove outdated information once in every few months.
One of the latest tips on feeding memory is from Thariq (a technical member of Claude code at Anthropic), where he gives a prompt that we can feed on chatbot to create memory based on our responses. I tried with ChatGPT and Claude, the responses after the interview was so much better. Once memory is updated, you can add custom instruction based on more questions, and even custom skills for each project. The possibilities are endless.
Understanding and leveraging recent trend of AI owning compute.
If you have been following prices of tech and gadgets you would have noticed that prices of all electronics have increased, as a result of AI tech companies buying and owning compute. It’s possible for chatbots to do agentic work through just chat interface alone. Grok build is one of the best example for this. To start using Grok’s agentic abilities you should have super grok subscription.
If you go to automations which is visible on your left hand side bar menu on grok.com . You can leverage automations as a means to complete a task when you are away. One way I am using is to generate and curate an personalized new feed on topics I am interested in.
The image shows a lot of templates, like to check for investments, to respond to mail and so much more. Before age of AI, one had to rely on compute like VPS, and had to know coding. Now AI can understand our intension and goal that are trying to do, and build skills around our final outcome, and trigger that automatically as long as we have grok application in our mobile phone, even if our PC is power off, the my automation setup delivers my news feed to my email daily at particular time. You can connect your mail by just tweaking options in settings, no need of API key, no manual work, just login to the connector with your mail, and allow grok to access your mail.
Pro tip 6#:
Staying on top of trends, and using AI features, not limiting to just one AI, allows us to do more than just sticking to one. Sometimes it does help us to try out and explore new things, new AI chatbot features and Models. Coming to the topic of Model, here is our final tip
Having Model IQ helps (final pro tip #7)
Model IQ is not a term that I came on my own. It’s something that was taught on class by Jason Haddix. I struggled to understand this concept, but it’s all about understanding abilities of the model we are using. This comes from lot of usage, observing output delivered by the model, and comparing it with claims of other people online, either through X, or through official blog post that gets released. Usually when opus 5, or any model gets released, there is lot of discussion that happens online, and combining that information through the lens of our experience on model usage helps.
One example would be frontier models like opus, usually get to do extra work that is usually not defined in our workflow, or skill. When we say do changes on this part of code or follow this, front tier models reads adjacent code, and directory structure to make more informed decision and goes extra mile for us. What this means is that even though front tier models can find 0 days and n-days, personally if I am using AI model for cyber work I should know when to stop, else I will be burning my tokens. Knowing what to do, how much to do and when to stop comes from experience and expertise that is built by constant learning, questioning, analysing and observing patterns, and going extra mile that few people miss.
To understand or to have model IQ, it does help to listen to people who elite in their profession. On the talk of building hackbots, Jason Haddix shares that not using max effort can sometimes hinder the ability to run agents for extended period of time, unless we don’t use Claude max.
My personal experience with using models to audit cloud applications, is that using models like sonnet, Gemini , and even GPT is helpful, then using heavy models like OPUS. Sometimes AI evaluates and expects lot of evidence for a security control than it’s required. And AI gives generic responses unless we steer it with hard fundamental questions that auditor needs to ask and understand during audit process. Sometimes taking a step back, understanding how control is implemented, and asking project team, reading Microsoft documentation, helps me understand more about security control than asking chatbot with latest and greatest model. Maybe it’s the way I am using model, and maybe my skill allows me to use smaller models effectively and it does take different set of skills to make heavy models work effectively without burning tokens. I am using heavy models to understand core concepts of azure, find any contradictions between official documentations, and I am using less AI to understand the audit controls, and little less AI for finding evidence for that security control.
Now if Model IQ is something that interests you, you can start by understanding the relationship by effort setting on AI chatbot. You can do more googling and ask questions like does using more reasoning affect temperature setting of chatbot? and apply similar questions to understand how to utilize memory, in chatbots.
Conclusion
If you have made it to the end of the blog, congrats, and give yourself a pat. If there are only two things you have to take from this blog, and the two core ideas are
1. Try and experiment with AI capabilities and chatbot features. See and experiment what fits your needs, and what works for you and if time allows try to experiment with others.
There are a lot you can do. You can connect with connectors like notion and prompt chatbot to take notes. Or you can connect your social media using connectors like “Apify” and automate your LinkedIn post generation or even apply to jobs automatically. The purpose of the blog is to just expose you to the ideas of countless possibilities.
2. Try using AI as sparring partner to challenge your understanding, assumption and to enhance your mode of learning rather than using AI to get your work done.
I know it’s very tempting to use AI credits to do your work, and get things done. And not always we can do work manually, but if you ever get a chance to do work manually, and learn things, don’t miss that opportunity. There are lot of study online that people who use AI blindly, and who don’t question or think, AI will make them worse at work and will degrade their ability to do work effectively. Cognitive decay is real, and AI can make bad students worse. I learn this the hard way. Asking questions is the one of the effective ways to learn. Asking specific questions is a sign of intelligence. Rather than simply saying I did not understand this whole topic, trying to prompt into AI about specific section of the topic is helpful. Asking AI to address your fallacy, and understanding is a great way to start. And if all this is something that is new, you can create a skill with just a single prompt that makes AI your sparring partner during studies, instead of answering questions, if gives you reference links online, and asks you questions.
Importance of understanding concepts at core level through leveraging various mental models
In words of Andrej Karpathy, you can’t outsource understanding.
In a world where everything has become a essential commodity (AI can write faster and better words, AI can generate lot of ideas, code, and find bugs), the ability to reason, ask questions and rationalize is truly a skill. You can start by asking following questions to AI like
From first principles explain me how SQL injection works, what are conditions required for SQL injection to work and when is it considered as N/A?
What happens when someone tries " or 1=1 --?
If you have noticed, we have tried to apply 3 mental models in our prompts and these are:
- First principles thinking
- Second order thinking
- Invarient
I highly encourage you to watch and google these mental models. Using first principles approach Elon Musk was able to reduce cost of rockets and their components, during his venture on SpaceX.
First principles thinking is the concept of understanding an idea, from its core roots, breaking an idea down to its fundamental components. This thinking helps us to understand things deeper.
Second order thinking simply means to reason out what happens next when XYZ occurs. This consists of reasoning and analyzing the consequences of consequences, rather than stopping at the immediate result.
And you can apply thinking principles like Invariant (testing whether your reasoning still holds when an assumption/condition changes.). You can apply this mental model by asking questions like "What remains true even when other or required conditions for this attack changes?”
So, to answer my own question that I asked at start, if elite as using to replicate their skills, and build systems around their expertise? how am I going to use AI to learn and upskill myself?
The answer is through applying thinking principles like first principles, second order thinking, invariant, and through struggling voluntarily during learning slope, in order to make most out of learning. Learning is definitely a slow process, and giving oneself enough time, and grace to understand things is very important. Not rushing is also important IMHO.
Now we have reached the end of the blog. If you have not set up your projects, chat instructions, memory and skills the way you wanted, it’s your sign to start working on them, and to set it up as per your preference, and especially in a way you want it to be.
It’s also highly recommended to build your own study skill on AI chatbot based on your learning goals, learning style, immediate certification goals, based on mental models described in this blog.
This blog was typed by completely hand, and ChatGPT’s image generation was used to illustrate ideas wherever it was needed to retain attention of reader, and for easier understanding of ideas discussed in the blog. Writing part was completely done by human (me), without help of AI.
