Like Warthog said earlier, the deck rot issue is most likely not related to the O-rings. The rot is due to the core being exposed to moisture, either through the screw holes or the hole cutout itself. The only way to be sure that those are protected is to completely remove the hatch body, scrape/sand off any old sealant around the hole (I use a sanding disk in a Dremel) until you get down to clean, solid wood core, and then seal the whole cutout with several coats of epoxy.
For the screw holes, over drill the holes to a size significantly bigger than the diameter of your fasteners, making sure not to break through the bottom layer of glass. An easy way to do this is to measure the thickness of the core, then put a piece of tape on your drill bit to make sure you don't drill deeper than the core. If you do break through, tape off the bottom of the hole with a good strong tape (I prefer Gorilla tape). Seal the edges of the hole with the same epoxy you're using to seal the cutout, a Q-tip works well to get into the hole. Then, while the epoxy is still curing, fill the over drilled holes with thickened epoxy, dripping it in slowly to ensure you don't trap any air bubbles at the bottom of the hole. You can also poke it with a toothpick to release any air bubbles, adding more thickened epoxy afterward. Once the epoxy has fully cured, drill a hole in each epoxy "plug" to fit the fastener you're using to install the hatch, again making sure not to drill all the way through the plug with the tape trick.
Your core is now adequately protected from water intrusion, which will protect the wooden core material from rot. Waterproofing the hatch installation and maintaining the O-ring will not do this reliably, at least not for very long. However, doing so is important to keep water out of the tank compartment and surrounding foam, in order to protect the tank from corrosion and avoid accumulating water.
To install the hatch, insert it into the cutout bedding it with a sealant such as LifeCalk. 5200/4200 sealant/adhesives are NOT necessary in this situation, since the hatch is being held in mechanically by the fasteners, so you just need the sealant properties without the adhesive which will make it difficult to remove in the future. I would not recommend using a silicone-based sealant, as if you do need to remove the hatch in the future the remaining cured silicone will prevent you from being able to apply new sealant. A thin bead around the hatch will be sufficient. Tighten the fasteners so that the hatch is secure and the sealant isn't "oozing" out the edges of the hatch body. A small amount of squeezeout is acceptable, but too much risks creating an incomplete seal as there will be an insufficient amount left under the hatch body. For easy sealant cleanup, tape can be laid out around the installation area of the hatch, and the edge of the hatch (dry fit before final installation) traced out on the tape. Cut along the tracing with an Xacto knife, and remove the tape underneath where the hatch will seat. Then, upon final insulation, any squeezed out adhesive can be easily removed with the tape WHILE STILL WET. If it is allowed to cure it may bond with the tape and make removal very difficult.