AISA (aisa.to) AI Fluency Assessment
Measure AI Fluency with a chat, then prove and improve it.
Start new thread

What is BAD vs. GOOD AI judgement?

by
•

Would love to hear thoughts.
Here's my perspective.

I'm against AI slop, but I'm not against AI. I believe AI is a great tool but it also needs greats pilots to use it well. This is what I'm working on. AI Fluency.

Biggest marker of AI fluency is AI judgement. Where and when to use AI and where and when not to.

BAD AI JUDGEMENT

  • AI avatar job interviews. A robot face asks you scripted questions and scores your vibe. No human in the loop. You're not being interviewed — you're being processed.

  • AI-generated "thought leadership." Copy, paste, post. Zero lived experience, zero original thought. It's just AI slop in a blazer.

  • AI-written emails to people who matter. Your client can tell. Not because the grammar's off — because the warmth is.

  • VS

GOOD AI JUDGEMENT

  • Using AI to actually look at what a candidate did. Not keyword-matching a CV — pulling apart their portfolio, cross-referencing their claims. You still decide. You just decide better.

  • Using AI to draft, then making it yours. AI beats the blank page. But the final thing sounds like you and says only what you'd stand behind.

  • Using AI to poke holes in your own plan. Feed it your strategy and ask it to break it. Not to decide for you — to make sure you're not being lazy.

What do you think?

204 views

Add a comment

Replies

Best

The poke holes in you plan, example really resonates. Ai is most valuable when it improves our judgment , not replaces it.

 really agreed with you . but my point is to add that interface matters also.

   can you elaborate a bit Riley?

 Agreed - though it takes some AI fluency to be able to tell that and have a nuanced judgement.

The poke holes in my plan example is one of the strongest. using AI as an adversarial thinking partner feels much healthier than asking it to make every decision.

 Something I learned is to "red team" it.
And with agentic systems what I also frequently and up doing is ask to create multiple agents who each hold a perspective and push from their own angle, and finally a consolidation layer based on certainty score.

For me the biggest signal is ownership. if you are willing to put your name behind the final decision and can explain how you got there, AI can be incredibly useful.

 100%.

It doesn't matter if AI created it with my prompt or I did it without AI. If I'm sending it, I have to own it.

The real skill is not getting AI to answer. Its knowing which questions you should not outsource.

I am not sure AI written emails are always bad judgement. sometimes the best use is getting the structure right then putting your own voice back into it.

 Agreed. Again depends on the email and context I guess.

i think AI judgement really shows up in the moments where using AI feels convenient but is actually the wrong choice Testing those edge cases could reveal more than simply measuring how well someone uses AI.

I think the interface matters too. if a product makes AI feel like the default answer to every problem it subtly trains users to delegate things they probably should not.

 This is especially true with younger people / students. We did a 2K study looking at scores across different domains, "critical thinking" which is one of the key markers of how AI fluency is measured, is lowest in students.

Approach the findings with a grain of salt of course, as this is a self selected sample at the end of the day rather than a peer review paper, but in a world where we have very limited quantifiable research, it's saying something.

Actually let me dig up the article we published on it...

Here it is:

A good AI judgment will always be backed by data, always be.. and factual data, not generated one.. and it would be building reasoning with data physics and maths.. not decisions..till AI can claim with 100% certainty that the data sources are exhaustive.. else the judgment is incomplete.. Bad AI judgement would be in many ways probabilistic and assuming..

I use AI constantly for drafts, but i have noticed the moment i stop editing and just accept the first version, my writing starts sounding like everyone else.

 Correct. It's an averaging engine in some ways. Have you tried feeding it your writing though, and ask it to categorically try to quantify which makes it unique to you, and then take that as it's reference writing?

Avoid using AI to support creative projects, but not to do the creative work and replace the creatives themselves. It can go horribly wong!

 Can't say I fully agree with that statement I'm afraid Peter. AI can absolutely, and beautifully, be used to "support" creative projects.

Creatives need to adopt to thinking in a world where AI is the norm. We had 3 years to catch up to this idea now.

 Hey Ozan! I completely messed up my comment. I mean to suggest use AI to support creative projects, but avoid using it to do the creative work itself! Doh!

Some of the marketing materials I've seen where they use AI instead of graphic designers is absolutely awful.

12
Next