Access is denied. Check that the Default Content Access Account has access to this content.

I had the following error on an Extranet site that I setup recently:
The start address <https://abc.def.com> cannot be crawled.
Access is denied. Check that the Default Content Access Account has access to this content, or add a crawl rule to crawl this content. (0x80041205).

I had read on other blogs that the trick was to login to the Index Server as the MOSS Search account, check that you can access the site, and verify the proxy/exclusions had been setup correctly. This normally works, but it didn't for me this time. And it certainly wasn't permissions as the search account had full read access to the site through the Application Policy interface within Central Admin.

In the end, I found that a registry change and a reboot of the server was required to fix it. See http://support.microsoft.com/kb/896861. I used Method 1 (Disable loopback check) to fix the problem. That is:

1. Click Start, click Run, type regedit, and then click OK.
2. In Registry Editor, locate and then click the following registry key: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Lsa
3. Right-click Lsa, point to New, and then click DWORD Value.
4. Type DisableLoopbackCheck, and then press ENTER.
5. Right-click DisableLoopbackCheck, and then click Modify.
6. In the Value data box, type 1, and then click OK.
7. Quit Registry Editor, and then restart your computer.

I hope this helps. It frustrated the living daylights out of me at the time.