hyperref does not work when using imakeidx

[xiehao]

The hyperlinks of page numbers in the Index failed to work when using imakeidx package. The reason is that the hyperref package should be loaded AFTER imakeidx. So is there a way to do this?

EDIT:
I have tried to pass nohyper option when loading tufte-book class, and call \TufteLoadHyperref right AFTER loading and configuringimakeidx package. However, some side effects appears, e.g. the font size of captions in margintable and marginfigure becomes weird.