Hype's video element is basically only a vanilla HTML5 <video> tag, so unless there's something specifically new in the spec for chapters, I don't think there's a quick built-in way to do this. But it should be entirely possible with some level of custom code.
Looking at it, there's often 3rd party libraries used to display chapter markers or at least a little bit of code that takes VTT files and presents them in a clickable fashion.
Generally you can include those libraries by dragging them into Hype's Resources Library, along with the video file and anything used (like a VTT). Then you can use these in an empty rectangle element via the 'Edit > Edit Element's Inner HTML' menu or sometimes via the Head HTML from the Document Inspector if needed. The files you added can be referenced via the ${resourcesFolderName}/filename.vtt type of syntax where "${resourcesFolderName}" is a magic variable that gets replaced with the correct folder name on export.
An example of using .vtt subtitles is here:
If there's a specific library or way you're familiar with making chapters for HTML video, feel free to reply back and we can offer more tips/guidance.