This man is the head of the yakuza gang that Yotaro was once a part of. He likes to listen to Yakumo-sensei's rakugo.
Spoiler He is eventually arrested by the police in the series. It is also hinted that he might be the biological father of Konatsu's son.