[arabesc]

If XySubFilter is not smooth, you are likely over the rendering time in madVR

It's not the case. Frame rendering time is about 4-15ms. A movie playback is completely smooth until first subtitles appear. After that the playback is jerky regardless of subtitles. And it doesn't affect the frame rendering time.
I've tested it many times. The first subtitle rendering just breaks something.

or running out of memory.

Do you mean video memory? I have an iGPU and lots of RAM.

There is a trade quality for performance checkbox in madVR related to subtitles to help with this. XySubFilter uses a fair bit of memory and processing power. So you have to account for this in madVR. If you own a GT 1030 or GT 1050, it could really struggle with XySubFilter

If XySubFilter spoils playback even when there are no subtitles on a screen, I think there is definitely something wrong with it.

madshi provided the last update to XySubFilter, but it is not being actively developed. It is his preferred subtitle filter. There is nothing to update at this point, as nothing new has happened with subtitles over the past couple of years. Some tweaks to HDR subtitles have been considered. But that's about it.

Here is an answer to my report that confirms some issues with the XySubFilter. BTW, your answer is there too.

You can use another DirectShow subtitle filter by adding it as an extra filter in DSPlayer and specifying it as a subtitle filter. 

Thanks.