Fixing Search removing all machines from farm

Scenario: Search is up and running in a 3 server farm. SP1, SP12 and SP14. Remove all 3 machines from farm and join back again. Search is broken. See in ULS logs There are no addresses available for this application.

First how to set up the Repro Environment

  1. Run the Create Search App powershell on the Central Admin machine, this way the Admin Component will reside on the CA machine. The script is available on SharePoint Brew web site.
  2. Go to CA, and move the Crawl Component to second machine via the UI in Modify Topology. ( There are 2 ways to do this, You “can create new” and “delete old” in same step or just move the existing crawl component by choosing server name in pick list on edit properties of Crawl component, then apply topology changes and wait) if you get an error you will have to follow Troubleshooting Step C below)
  3. Create 2 new Query Components so that each machine has a Query Component. click Modify Topology and click Add New Index Partition and Query Component.
  4. Run a full Crawl and wait till complete and test to make sure Search works on at least one site. Make sure the Proxy is set for the site you are testing on.
  5. Run this command to on all 3 machines to disconnect from Farm – psconfig -cmd configdb –disconnect
  6. Uninstall and re-install sharepoint
  7. Run this to join Farm – psconfig -cmd configdb -connect -server sp1 -database SharePoint_Config -passphrase Katar1na
  8. Then run the wizard – psconfigui.exe
  9. Search is now Broken

Here is how to fix, basically you need to find GUID’s, run commands, restart app pools and wait for a few minutes and it will start working. This is proven for the scenario where your reset index and move all search to 1 machine. Details as follows. The Search will appear online in Central Admin but it is not. When you run a search from a site you will get an “Internal server error exception” with a Correlation ID. if you try a Crawl it will stall out so do not try until it is fixed with this procedure.

Check that the Administration Component is Active

Start the disabled Search Service in Windows

Run this on each machine that will participate in search.

$SSI = get-spenterprisesearchserviceinstance -local
# Start search services for on each machine for SSI
Start-SPEnterpriseSearchServiceInstance -Identity $SSI

Run this to get the SP Service Instance GUIDs of these 2 apps on the correct server. Note server name in results and make sure to pick that GUID.

Get-SPServiceInstance | fl id, Server, typename

Results
Id : 5f0a1d17-7e07-49cd-8b85-dcd6ac451d40
Server : SPServer Name=sp1
TypeName : Search Query and Site Settings Service

Id : fc131bd5-21c0-429e-9ed4-2ca640e9f460
Server : SPServer Name=sp1
TypeName : SharePoint Server Search

 

Start the Service Instance (using Guids from above step)
start-spserviceinstance <Your GUID Here for Search Query and Site Settings Service>

Get-spserviceinstance <Your GUID Here for Search Query and Site Settings Service>  # to check status, should say online

start-spserviceinstance <Your GUID Here for SharePoint Search Server>

get-spserviceinstance <Your GUID Here for SharePoint Search Server> # to check status, should say online

Add a new Crawl Component

Troubleshooting

A. Wait for things to provision, and check to make sure the Service Application Proxy is assigned to your web apps you are testing on.

B. Un-check and re-check the service proxy for Search Service Application, then wait few minutes and test searching

C. If the crawl component becomes stalled or in recovery status  for a while, delete and re-create it as follows

$ssa = get-spenterpriseSearchServiceapplication
$ssa.querytopologies
# We see two query topologies, one active and one inactive.
# Grab the ID for the inactive query topology.
#Run this :
$qt = get-spenterprisesearchQuerytopology –id “38c19e23-ab3e-43f1-a3c1-0faa708e3ddf” –searchapplication $ssa
$qt.delete()

D.  net stop osearch14   or   net start osearch14 or run together    net stop osearch14 && net start osearch14

 

Details while running the above commands

PS C:\Users\sp1admin> Start-SPServiceInstance fc131bd5-21c0-429e-9ed4-2ca640e9f460

Results
TypeName        : SharePoint Server Search
Description     : Index content and serve search queries
Id              : fc131bd5-21c0-429e-9ed4-2ca640e9f460
Server          : SPServer Name=sp1
Service         : SearchService Name=OSearch14
Role            : None
QueryComponents : {45953c5e-4655-4ba0-9d26-8047a47095c4-query-12}
CrawlComponents : {45953c5e-4655-4ba0-9d26-8047a47095c4-crawl-2}
AdminComponents : {45953c5e-4655-4ba0-9d26-8047a47095c4}
Status          : Provisioning

 

PS C:\Users\sp1admin> $a = get-SPServiceInstance fc131bd5-21c0-429e-9ed4-2ca640e9f460
PS C:\Users\sp1admin> $a
TypeName        : SharePoint Server Search
Description     : Index content and serve search queries
Id              : fc131bd5-21c0-429e-9ed4-2ca640e9f460
Server          : SPServer Name=sp1
Service         : SearchService Name=OSearch14
Role            : None
QueryComponents : {45953c5e-4655-4ba0-9d26-8047a47095c4-query-12}
CrawlComponents : {45953c5e-4655-4ba0-9d26-8047a47095c4-crawl-2}
AdminComponents : {45953c5e-4655-4ba0-9d26-8047a47095c4}
Status          : Online

 

PS C:\Users\sp1admin> Get-SPServiceInstance 5f0a1d17-7e07-49cd-8b85-dcd6ac451d40

Result
TypeName                         Status   Id
——–                         ——   —
Search Query and Site Setting… Online   5f0a1d17-7e07-49cd-8b85-dcd6ac451d40

 

 

PS C:\Users\sp1admin> Get-SPServiceInstance

Result
TypeName                         Status   Id
——–                         ——   —
Managed Metadata Web Service     Disabled 505bc829-a73d-4c1a-bf1f-e5f2b9a896bc
User Profile Synchronization … Disabled 6a7aacfa-2c05-42d9-bca5-3a662b73268a
Business Data Connectivity Se… Disabled f9623df5-86da-400a-b5cd-b70cf67e1fa3
Secure Store Service             Disabled 64a3264e-5957-4a86-a63e-ab1b758179a0
Claims to Windows Token Service  Disabled 2a5854d3-d6ab-416c-97aa-75ca09e06902
Microsoft SharePoint Foundati… Online   ab2a1bed-8140-43b4-a9e2-114741f96293
PerformancePoint Service         Disabled e257161f-2f83-4f5f-a729-b440c9e2a69a
Application Registry Service     Disabled ce3118d0-823a-4d1b-abd8-1724fd6d7e28
Microsoft SharePoint Foundati… Disabled 184fdc2d-7116-4a85-a60f-bfd895c3d244
Visio Graphics Service           Disabled 6f4dec89-f8e0-4f57-9891-4d8dfbf23a96
SharePoint Server Search         Online   f431d93a-e167-4424-b96c-2a9105ce2ed6
Document Conversions Launcher… Disabled 14dc20d2-20ab-4fb5-badc-96822d87e5d4
Document Conversions Load Bal… Disabled 9e532ad3-7c76-46a9-a389-f2bdff6444c4
Search Query and Site Setting… Online   a11539ca-e23c-459c-a0b0-74c90884fe71
Web Analytics Web Service        Disabled 49521334-b2ac-4f10-968b-4f99302ad365
Microsoft SharePoint Foundati… Online   0582d1cb-30c8-4666-8b0d-e3a36a6ad01e
Central Administration           Online   1c4526bb-df9c-49d2-a2f1-5971f52ca46f
Excel Calculation Services       Disabled 55352002-c88c-4b89-b8a1-c87806bb278b
Microsoft SharePoint Foundati… Online   277769da-50fc-44e7-b498-f71e486d522b
User Profile Service             Disabled 3a2c3901-ce0f-4dba-a7a4-fc0e0e1228ad
Access Database Service          Disabled e1d18074-da5e-4783-88f2-2a47bc47f53e
Project Application Service      Disabled 4a7aa8f2-2a0a-46fe-a54a-9b499b51dad3
Microsoft SharePoint Foundati… Disabled d46878f0-3ae4-4306-a139-2ec7a64b1dc1
SharePoint Foundation Search     Disabled a53791df-3ef9-4347-a2a2-52e2d16b6703
Lotus Notes Connector            Disabled 6fc54fb7-8981-46c2-a34c-623ed4cc797a
Web Analytics Data Processing… Disabled 29716b7f-8acb-4075-b3fc-e69405c62308
Word Automation Services         Disabled dd8f0f31-e289-4981-a566-555913fe8028
Managed Metadata Web Service     Disabled a307866a-d625-4f9f-bc71-e4c5f6d619ec
User Profile Synchronization … Disabled 62903dc1-2cae-466a-962d-1563acfb15d2
Business Data Connectivity Se… Disabled abea5704-39cc-4eea-be58-f29db99b7491
Secure Store Service             Disabled fa75f7be-039f-4ba7-a1d5-33d77d3c0e81
Claims to Windows Token Service  Disabled 3fef51da-9f90-43ff-80bf-5f90ad187505
Microsoft SharePoint Foundati… Online   5635fa0e-5ce2-488f-958a-1b03c0708806
PerformancePoint Service         Disabled 93e3811b-c0fa-4a0b-a8c7-7ac4c8c7c3f7
Application Registry Service     Disabled 48ba2bee-1b32-475b-b4c4-6b51167a3183
Microsoft SharePoint Foundati… Disabled 52b51bc7-90f8-4906-b71b-793920a84485
Visio Graphics Service           Disabled 42144652-253c-477e-acbe-50a1ba12e20f
SharePoint Server Search         Online   29ddcf6f-25b0-4ade-9a1e-ea655e76b404
Document Conversions Launcher… Disabled b30d6457-c997-4ca2-a730-c5fb78dc1a3c
Document Conversions Load Bal… Disabled 18740351-213b-4e2d-a331-4cf788182264
Search Query and Site Setting… Online   64734e6d-2fad-4130-a6cb-0b9b7ff1eade
Web Analytics Web Service        Disabled 46cd332f-b05d-4c86-b4f0-64be52182fe7
Microsoft SharePoint Foundati… Online   723815a3-5156-44c0-97fc-a79e008d2a9b
Central Administration           Disabled b5fe91fe-77dc-484c-836e-e1bfd8b51595
Excel Calculation Services       Disabled cc3602e6-4e5d-4dbe-9094-3c88abd3f22e
Microsoft SharePoint Foundati… Online   5a24d7b6-e473-4087-8858-93787cc9bf69
User Profile Service             Disabled b8c71aa9-3fd1-4d65-9731-84bbde70c3cf
Access Database Service          Disabled f8fc5c31-60ff-4967-b1ac-781e6dabc5d6
Project Application Service      Disabled 70b4a8f9-957e-4701-8521-a2d9875eec86
Microsoft SharePoint Foundati… Disabled 35ce779a-0b7e-4c48-99b7-78099d0cf084
SharePoint Foundation Search     Disabled 198e8879-de3b-4c43-8f4c-85ecffcb13e3
Lotus Notes Connector            Disabled 729dbe1f-72ea-4440-835e-672428f71dfb
Web Analytics Data Processing… Disabled a3eee77f-8cee-4eae-8f19-862c44c99802
Word Automation Services         Disabled 69fac58f-70c9-46ef-8773-7e5862d2677a
Managed Metadata Web Service     Disabled 78a367c5-c479-46ba-9f15-1aacaa61db15
User Profile Synchronization … Disabled 42fdc2a2-7e6c-4204-81e0-3865fa1d2436
Business Data Connectivity Se… Disabled 7df74bbd-cec0-4b36-8349-b25ddb27d607
Secure Store Service             Disabled 1b875966-f493-4cf3-b4bb-e0aa39bd9f30
Claims to Windows Token Service  Disabled 1679f2c8-94a2-4ce5-bbac-5765235204f6
Microsoft SharePoint Foundati… Online   9a40d4f6-d168-4692-abf3-3eadfa91700c
PerformancePoint Service         Disabled 05ebadc3-291d-484e-8fb2-e3c04f63cfa2
Application Registry Service     Disabled 7848d7ec-da22-4d6e-a166-5f253dc8d80f
Microsoft SharePoint Foundati… Disabled 308bac96-0b0c-42d3-b37c-db783b4d64c7
Visio Graphics Service           Disabled d9d66646-968a-4cff-a97f-04f19cf168e8
SharePoint Server Search         Online   e8c80b12-5d58-41ad-b435-c6c2e6a8948d
Document Conversions Launcher… Disabled 94efb23b-5858-4fd7-8fc4-dbf7bf3b229d
Document Conversions Load Bal… Disabled d6b8d6c8-eae3-4ebf-b029-ca22ef31251b
Search Query and Site Setting… Online   86342999-7b96-4ee5-9737-2836fa3f0bc8
Web Analytics Web Service        Disabled 538e5e07-d698-4760-b680-f2e51c3e44b9
Microsoft SharePoint Foundati… Online   cad0f991-ec41-41d8-adb9-03fe35c514e1
Central Administration           Disabled 88a7033d-d7d5-42b5-a3f4-0ea8389ef4ca
Excel Calculation Services       Disabled 7e39851e-4d45-455c-94e9-24ecf069b84a
Microsoft SharePoint Foundati… Online   0b55df23-b974-43a0-9dc4-7b1e92d8e4aa
User Profile Service             Disabled c9c69183-93fa-41d9-ba08-a96a99e33bce
Access Database Service          Disabled b5128a09-acb6-474a-aaf5-4b9307bde539
Project Application Service      Disabled deead7ed-85fb-4ec4-86d0-1e20b2c00652
Microsoft SharePoint Foundati… Disabled e13656b2-ad08-4a88-a891-e04092db2122
SharePoint Foundation Search     Disabled 4dcb51a8-d210-470d-9c93-d96d4a6529c6
Lotus Notes Connector            Disabled 30c0fbcd-458b-457c-9ae4-8d467155d38f
Web Analytics Data Processing… Disabled 9907589f-d81b-4b8a-a0ca-f54f0f87a56d
Word Automation Services         Disabled 2ca56250-14b2-4214-8208-e2207860466c
PS C:\Users\sp1admin> $s = Get-SPServiceInstance e8c80b12-5d58-41ad-b435-c6c2e6a #Get Guid from above list
PS C:\Users\sp1admin> $s

Results
TypeName        : SharePoint Server Search
Description     : Index content and serve search queries
Id              : e8c80b12-5d58-41ad-b435-c6c2e6a8948d
Server          : SPServer Name=SP14
Service         : SearchService Name=OSearch14
Role            : None
QueryComponents : {a6a30de4-0a70-4f80-a620-6b9e9cb28533-query-3}
CrawlComponents : {} There is no crawl component, it will get re-provisioned with our fix above
AdminComponents : {} #Note it is missing and could be disaster, you can try to assign Admin component from the CA Box using PowerShell script in Appendix A below. If it fails there is no recourse except to delete and re-build Search
Status          : Online

 

This is how to fix web.config for “unable to connect to the Search Service” error

trustlevel in web.config, change to wss_minimal or wss_medium or wss_full

C:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\14\config -there are 3 files in folder, if custom file is in there, it has to be put back. The search request was unable to connect to the Search Service. #first error when run query in ui set callstack = true custom error =Off O must be caps

How to delete the entire search service app and start over.

PS C:\Users\sp1admin> $ssa = get-spenterpriseSearchServiceapplication “Search Service Application”
PS C:\Users\sp1admin> $ssa

Name : Search Service Application
Id : a47b6154-cc5b-4354-9b82-c3f06c946254
ServiceName : SearchQueryAndSiteSettingsService
QueryTopologies : {96cd3ecb-9b67-4301-817d-7c9af136573d}
PropertyStores : {Search_Service_Application_PropertyStoreDB_1a1fb46addad479e980595f95a12c90
7}
CrawlTopologies : {7e412bff-5db1-49f3-b7b7-150023f64068}
CrawlStores : {Search_Service_Application_CrawlStoreDB_6ab309b6495b4be9913d4291f9b7b3d0}
SearchAdminDatabase : SearchAdminDatabase Name=Search_Service_Application_DB_8e6aba754f6f4385bd9d
b6cbd45dd8fa
Status : Online
SearchApplicationType : Regular
DefaultSearchProvider : SharepointSearch
Properties : {Microsoft.Office.Server.Utilities.SPPartitionOptions}

PS C:\Users\sp1admin> $ssa.delete

MemberType : Method
OverloadDefinitions : {System.Void Delete()}
TypeNameOfValue : System.Management.Automation.PSMethod
Value : System.Void Delete()
Name : Delete
IsInstance : True

This is the command to do the actual deletion
PS C:\Users\sp1admin> $ssa.delete()

 


The balance is just some remnants that we ran not sure what they do

PS C:\Users\sp1admin> $qt = Get-SPEnterpriseSearchQueryTopology -id 965df711-24d4-4fee-8fcc-4ff247d0
a1e2 -SearchApplication $ssa

PS C:\Users\sp1admin> stsadm -o provisionservice -action start -servicetype “Mic
rosoft.Office.Server.Search.Administration.SearchService, Microsoft.Office.Serve
r.Search, Version=14.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c” -s
ervicename OSearch14
The local farm is not accessible. Cmdlets with FeatureDependencyId are not regis
tered.

 

PS C:\Users\sp1admin> $searchapp = Get-SPEnterpriseSearchServiceApplication “RAMQSearch”

___________________________________________________________________________________________________________________________________________________________
PS C:\Users\sp1admin> $searchapp
Name                  : RAMQSearch
Id                    : 531fb4e2-97bf-4754-b71d-de58d77f4e00
ServiceName           : SearchQueryAndSiteSettingsService
QueryTopologies       : {2d946c8c-58cc-46f0-a3fd-4a1fd77b20bf, 0af28960-5b54-4ae8-9039-5619e9f536ce}
PropertyStores        : {RAMQSearch_AdminDB_PropertyStore}
CrawlTopologies       : {f32f83bf-7bc9-4a40-aa0e-06732f5f4b7a, 4aeb28cd-a5ca-4ef8-afb9-b4497e17c4f1}
CrawlStores           : {RAMQSearch_AdminDB_CrawlStore}
SearchAdminDatabase   : SearchAdminDatabase Name=RAMQSearch_AdminDB
Status                : Online
SearchApplicationType : Regular
DefaultSearchProvider : SharepointSearch
Properties            : {Microsoft.Office.Server.Utilities.SPPartitionOptions}

 

PS C:\Users\sp1admin>
$topo = Get-SPEnterpriseSearchCrawlTopology -SearchApplication $searchapp

___________________________________________________________________________________________________________________________________________________________
PS C:\Users\sp1admin> $topo
Id              : f32f83bf-7bc9-4a40-aa0e-06732f5f4b7a
CrawlComponents : {a6a30de4-0a70-4f80-a620-6b9e9cb28533-crawl-2}
State           : Active
ActivationError :

Id              : 4aeb28cd-a5ca-4ef8-afb9-b4497e17c4f1
CrawlComponents : {}
State           : Inactive
ActivationError :

 

Appendix A

#Try to assign the administration component again using PowerShell,
#retrieve the search application names on your farm using
#Get-SPEnterpriseSearchServiceApplication | select name, then specify the one you want to
#try out in the first line of code below:

#Used this at RAMQ after farm restore to initialize the admin component.
# Retrieve the search application names on your farm using Get-SPEnterpriseSearchServiceApplication | select name

$searchApp  = Get-SPEnterpriseSearchServiceApplication #-Name “YourSearchName”
Write-Host -ForegroundColor white Initializing the administration component
$searchInstance = Get-SPEnterpriseSearchServiceInstance -Local
$searchApp | Get-SPEnterpriseSearchAdministrationComponent | Set-SPEnterpriseSearchAdministrationComponent -SearchServiceInstance $searchInstance
$admin = ($searchApp | Get-SPEnterpriseSearchAdministrationComponent)

Write-Host -ForegroundColor white Waiting for the admin component to be initialized
foreach($i in 1..120)
{
sleep 1;
Write-Host -ForegroundColor white .
if ($admin.Initialized)
{
break
}
}
if ($admin.Initialized)
{
Write-Host -ForegroundColor Green “Administration component initialized properly”
}
else
{
Write-Host -ForegroundColor Red “Administration component failed to initialize after 2 mins”
}
#At the end of the script, you should either see a success or a failure, if you see a success, perfect. If not, please rerun the code in the past post (Thursday, March 11, 2010 9:52 PM) again and paste the output.
End

 

 

Tried to make the above solution work where we leave the Search Toploogy across 3 machines, this is the error in the ULS

g1j9    Exception    Internal server error exception: System.ServiceModel.FaultException`1[System.ServiceModel.ExceptionDetail]: Object reference not set to an instance of an object. (Fault Detail is equal to An ExceptionDetail, likely created by IncludeExceptionDetailInFaults=true, whose value is: System.NullReferenceException: Object reference not set to an instance of an object.    at System.Runtime.InteropServices.Marshal.ThrowExceptionForHRInternal(Int32 errorCode, IntPtr errorInfo)     at Microsoft.Office.Server.Search.Query.KeywordQueryInternal.Execute()     at Microsoft.Office.Server.Search.Query.QueryInternal.Execute(QueryProperties properties)     at Microsoft.Office.Server.Search.Administration.SearchServiceApplication.Execute(QueryProperties properties)     at SyncInvokeExecute(Object , Object[] , …    893925d7-ad47-4463-8203-274000ffb0bb
06/07/2012 22:39:18.94*    w3wp.exe (0x0714)                           0x0D30    SharePoint Server Search          Query                             g1j9    Exception    …Object[] )     at System.ServiceModel.Dispatcher.SyncMethodInvoker.Invoke(Object instance, Object[] inputs, Object[]& outputs)     at System.ServiceModel.Dispatcher.DispatchOperationRuntime.InvokeBegin(MessageRpc& rpc)     at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage5(MessageRpc& rpc)     at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage4(MessageRpc& rpc…). System.ServiceModel.FaultException`1[System.ServiceModel.ExceptionDetail]: Object reference not set to an instance of an object. (Fault Detail is equal to An ExceptionDetail, likely created by IncludeExceptionDetailInFaults=true, whose value is: System.NullReferenceException: Object reference not set to an instance of an object.    at System.Runtime.InteropServices.Marshal.ThrowExce…    893925d7-ad47-4463-8203-274000ffb0bb
06/07/2012 22:39:18.94*    w3wp.exe (0x0714)                           0x0D30    SharePoint Server Search          Query                             g1j9    Exception    …ptionForHRInternal(Int32 errorCode, IntPtr errorInfo)     at Microsoft.Office.Server.Search.Query.KeywordQueryInternal.Execute()     at Microsoft.Office.Server.Search.Query.QueryInternal.Execute(QueryProperties properties)     at Microsoft.Office.Server.Search.Administration.SearchServiceApplication.Execute(QueryProperties properties)     at SyncInvokeExecute(Object , Object[] , Object[] )     at System.ServiceModel.Dispatcher.SyncMethodInvoker.Invoke(Object instance, Object[] inputs, Object[]& outputs)     at System.ServiceModel.Dispatcher.DispatchOperationRuntime.InvokeBegin(MessageRpc& rpc)     at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage5(MessageRpc& rpc)     at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage4(MessageRpc& rpc…).    893925d7-ad47-4463-8203-274000ffb0bb
06/07/2012 22:39:18.94     w3wp.exe (0x0714)                           0x0D30    SharePoint Server                 Unified Logging Service

Advertisements