There are types that signal a non-verbal you are interested and when you go on that, you suddenly have no more.
may be that you have misinterpreted something. Or that he is too shy to flirt active with you. I can not know this.
Who sends to weak signals, must also expect to be not perceived. You can flirt very directly and then the opposite knows. In this respect, one can never really know if it is flirted or not - if the signals are too weak.
best ask. Excluding the body language is not enough. You can quickly interpret that wrong.