SharePoint 2010 Fast Disk Space Part 2 (Solution)

Plan

The solution for the issue would be to stop the service and clear the logs. Start the service back try to process the old click analysis. It should typically process those files and improve the link rankings.

Note: This will have an impact on the search results proceed with caution.

Backup

Backup the following locations to a different server or a drive.

:\FASTSearch\var\log\sprel\

:\FASTSearch\data\sprel\ (all folder below this)

:\FASTSearch\components\resourcestore\generic\clickthrough

Execution

Stop the sprel process deletes the logs, start the service back and run the SPRel process cmd and ensure everything works as expected.Test all the process in a Test environment before trying it in a live environment.

  1. Run the following command to stop the sprel process:
nctrl stop sprel walinkstorerreceiver
  1. Clean the data/sprel directory (Except config folder)
  2. Run the following command to start the sprel process again:
nctrl start sprel walinkstorerreceiver
  1. Confirm everything is ok by running the following command
spreladmin ShowStatus
  1. Start the processing by executing the following command
spreladmin StartProcessing
  1. Regularly check the status by executing the following command
spreladmin ShowStatus
  1. Once completed, status shows ready doing nothing: ensure the process is completed

after

Update

we faced issue while incremental crawl. The following error popped up and due to this, some search results were affected.

The Content Plugin received a "Processing Error" response from the backend server
for the item. ( ERROR: PostProcessBatch failure in processor SPCIDStorer. 
Traceback (most recent call last): 
\\ File "SPCIDStorer.py", line 63, in PostProcessBatch \\ File "TableSender.py", 
line 111, in send \\ WindowsError: [Error 3] The system cannot find the path 
specified: 'E:\\FASTSE~1\\data\\sprel\\cidstorer_backup/*.*' \\ [17]

Fix: Create an empty folder :\FASTSearch\data\sprel\cidstorer_backup

Results

The expectation is the logs shouldn’t grow at a rapid pace, the clickthrough archive should be processed and important the search service shouldn’t see any outage.drive>:\FASTSearch\components\resourcestore\generic\clickthrough (data pushed by the SP extraction job) should be empty if the process worked as expected. Monitor the process for a couple of days and ensure there are no errors.

Check the following location should be empty if the process worked as expected.

:\FASTSearch\components\resourcestore\generic\clickthrough
(data pushed by the SP extraction job)

Start an incremental crawl to ensure there is no impact on the results. Monitor the process for a week and ensure there are no errors in the sprel logs or crawl logs.

Ref Links:

https://blogs.msdn.microsoft.com/microsoft_search_bloggers/2015/02/10/sprel-consuming-more-disk-space-and-spreladmin-status-not-showing-as-ready/

https://social.technet.microsoft.com/Forums/exchange/en-US/a76fad93-35a4-4de9-8662-c4c7f8b24225/sprelexe-high-utilization?forum=fastsharepoint

http://blog.comperiosearch.com/blog/2011/07/12/learning-about-nctrl-disabling-fast-search-web-crawler/

https://technet.microsoft.com/en-us/library/ee943516(v=office.14).aspx

 

SharePoint 2010 Fast Disk Space Part 1 (Issue)

I recently faced disk space issue on SharePoint Fast 2010 Crawl server. We need to find out what’s consuming the space. If it is the index or any core components we don’t have an option other than increasing the space.

Investigate DiskSpace

Let’s begin the journey with identifying the files, we will use a simple tool (windirstat) to find the space consuming files.

windirstat

The initial results were showing that the core components are not the issue and all that’s consuming is the logs. I was surprised to see a SPRel log that was 27GB and growing in standard phase, Next big thing was query logs. we have a PowerShell script that cleans up the query logs that are older than 60 days.

  1. FASTSEARCH\var\log\querylogs\  (Safely can be cleaned up)
  2. FASTSEARCH\var\log\sprel\
  3. FASTSEARCH\data\sprel\worker

We realized something is going wrong with the SPRel process. Just to give you an idea of the SPRel service.

“SPRel is a clickthrough log analysis engine that improves search results relevancy by analyzing the entries that users click on in search result sets. In FAST Search server 2010, we can use spreladmin.exe to configure SPRel, to schedule the clickthrough log analyses, and to retrieve status information”

Investigate SPRel

Check if the service is running as expected. Command to check the status should be executed in command prompt

<drive>:FastSearch\Bin\spreladmin showstatus

before

The process is actually stuck and not working as expected. This has stopped working in 2013 and till now in 2016. I think we are in the right direction. Now we have to analyze the log to find the reasons. Here comes another problem log is too huge to open it in notepad(27GB). It’s a Handy tool that saved my life Large Text File Viewer no installation required doesn’t crash your CPU it lazy loads the file with continuous query mode.

Logs

The possible failures found in the log, Disk Space

Disk Space wasn’t enough to process

Impossible to schedule more targets (timeout 60s). 4 ready targets:

make_1032.urls_by_urlid.1_b1. Target was never started. Too large output for the workers. Input size : 153196411

make_1032.urls_by_urlid.3_b3. Target was never started. Too large output for the workers. Input size : 153952094

make_1032.urls_by_urlid.0_b0. Target was never started. Too large output for the workers. Input size : 153724399

make_1032.urls_by_urlid.2_b2. Target was never started. Too large output for the workers. Input size : 153472337

— stopping build.

The Server didn’t respond and it has failed.

systemmsg Reset of analysis completed.
systemmsg Analysis failed 10 times, will retry from start.
systemmsg sprelrsclient failed. stdout: Connected to ResourceStore stderr: Could not list resources.. The remote server returned an error: (500) Internal Server Error.

I will take some time to figure out the solution and will discuss in the next post.