*i've tested all this in my sharepoint 2013 but as i understand it works the same in 2010.
the 1st point we must understand is that the more we want to enable in an anonymous site the more we need to take down security. so lets start!
initially indexing only the admin site, keeping the anon site without Win Auth will index your most of your published pages, excluding pages you checked the checkbox in their properties names "Dont show is search".
when the search indices an item it also indices it's permissions, therefor anything your anonymous user cant see he wont see as a search result. so the 1st thing he cant see is the admin site (the non-anon), and therefor we need to index the anon site.
so now i want my list items from my list... how can i get that?
most simple solution is using the rest service (not search rest). that's great as long as you're satisfied with it. but you wont see the items in a regular search page/center nor publishing/other-complex-rich site columns.
and even here you can solve it by asking each single item with like this:
http://server/site/sub/_api/web/lists/getbytitle('list name')/items(item id)/fieldValuesAsHtml
for the usual full formatted or just the text:
http://server/site/sub/_api/web/lists/getbytitle('list name')/items(item id)/fieldValuesAsText
setting anon access to the my admin site partially, choosing list and libraries and setting private permissions to every library wont help here.
still i tried to fight it a little and the 1st thing i tried is to crawl the anon site. now in order for the crawl to be able to run in our anon site he need to log into it, so 1st thing is to allow Windows Authentication, a thing we usually disable in the IIS after setting up the anon permissions. this can be annoying since, at least in my dev machine, the "Search" part of the page, both of a search center, search page, or rest call, all ask for credentials if the Win Auth is enabled, i am still looking about it but currently after a crawl i disable it.
i am still looking either to cut out the auth request or enable crawl without it.
also even there in order to really see the items i needed to deactivate the feature “Limited-access user permission lockdown mode” in Site Collection Feature.
but then the anon user can get anywhere (and thats bad!) and inside all my libraries so again setting each library in separate and it started to be a mess.
so i went back to full anon, indexing the admin only, having as results pages only.
about the results its the same issue as above.
the rest needs 2 things to work, a special xml you can create with the PS in this blog (at the bottom), he also gives an overview : http://summit7systems.com/enabling-anonymous-sharepoint-2013-rest-search-by-entering-one-url-in-powershell/
note that even after that you need to allow imersonation in the web config :
<identity impersonate="true" />
i assume you also need to allow it in the IIS.
also not sure but maybe you need to add a valid xml header to the file <?xml version="1.0" encoding="UTF-8"?>.
now you can make search rest calls, usually with zero results.
i heard something about anonymous libraries but it sounds like the above giving anon permissions so i am not sure about it and still investigating.
btw - my final solution in this case is making a subsite and a page-layout for my items since in my scenario it's helpful.
IMO the search was mainly designed to non-anon sites, for them there is the standard REST api. after all even standard search like search center or WP cant return list items as results.
therefor if there really is a case where you MUST be using the rest in anon maybe its an architecture mistake? not cause you're mistaken but cause MS never designed it for that.
p.s. remember that after each change you need a full crawl again.