Exploring ShareFile’s ‘StorageZones’ Services

I was looking for more information on what makes a ShareFile StorageZone ‘tick’, and couldn’t find much that got into the nuts and bolts of this great feature. This post is intended to share some general information about the various StorageZones controller services, including their basic functionality, and some hidden configuration settings.

For the scope of this post, I’m going to focus on the three Windows services that are installed as part of a StorageZone v2.1 Controller. Each service is installed off the root of the IIS site as follows:

  • File Cleanup Service – Citrix\StorageCenter\SCFileCleanSvc\FileDeleteService.exe
  • File Copy Service – Citrix\StorageCenter\SCFileCopySvc\FileCopyService.exe
  • Management Service – Citrix\StorageCenter\s3uploader\S3UploaderService.exe

In each of these directories are the service’s .NET .config file, which can be modified to enable logging, and adjust hidden configuration settings. For example, if you open FileDeleteService.exe.config, you’ll see the following XML by default:

<?xml version="1.0"?>
<configuration>
   <appSettings>
       <add key="ProducerTimer" value="24"/> <!--Time interval in hours-->
       <add key="DeleteTimer" value="24"/> <!--Time interval in hours-->
       <add key="DeleteTimer" value="24"/> <!--Time interval in hours-->
       <add key="Period" value="7"/> <!--No. of days to keep data blob in active storage after deletion-->
       <add key="logFile" value="C:\inetpub\wwwroot\Citrix\StorageCenter\SC\logs\delete_YYYYMM.log"/>
       <add key="enable-extended-logging" value="0"/>
       <add key="BatchSize" value="5000"/></appSettings>
<startup><supportedRuntime version="v4.0" sku=".NETFramework,Version=v4.0"/></startup></configuration>

As you might have guessed, setting enable-extended-logging  to 1 will enable verbose logging after the service is restarted, writing to the specified logFile path. This setting is the same for the other services, and can come in handy when troubleshooting issues with a StorageZone.

In order to really understand what these services were doing, I decided to poke through the source code by decompiling the services’ assemblies using a free utility called DotPeek. Here’s a summary of what I found for each service’s functionality within a StorageZone Controller.

File Cleanup Service (FileDeleteService)

The name says it all here, as this service’s sole responsibility is managing data deletion from the storagezone storage repository. Since all of the data stored by ShareFile is in BLOB format, deleting a file through the ShareFile front-end doesn’t actually delete it from the storage; it simply ‘de-references’ the data, and marks it as ‘expired’.

This ‘expired’ data will remain in the storage repository until it’s ‘cleaned up’ by the File Cleanup Service. This is why if you look at a folder’s recycle bin, you’ll see the files are still listed and available for recovery until the configurable cleanup period lapses (7 days by default).

Citrix recommends configuring this cleanup period to match the backup schedule of your storage device so that data is removed shortly before or after they’re backed up. This design also allows for data to be recovered even if it’s not in the recycle bin, by using the “Recover Files” function in the StorageZone section of ShareFile’s Admin page.

Here are the .config extended settings for this service, along with their default values:

  • ProducerTimer = 24 Time interval in hours 
  • DeleteTimer = 24  Time interval in hours
  • Period = 7 Number of days to keep data in active storage after deletion

File Copy Service (SCFileCopySvc)

This service is what allows the StorageZones controller to communicate with ShareFile’s cloud infrastructure (by way of the ShareFile API), and allows users to upload and download files directly to and from a customer’s on-premise storage.

When a file is uploaded, ShareFile’s servers connect to the controller through this service to initiate an HTTP(S) POST request, allowing the data to be stored directly to the StorageZone. The service also converts files to and from the ShareFile’s proprietary format, converting files to BLOB data for uploads, and converting BLOB data back to the original file for downloads.

The service also has a configurable timer value (the default is 10 seconds key=”CopyTimer” value=”10000″ ) that controls how often retries are attempted for jobs that previously failed due to connectivity issues.

Management Service (S3Uploader)

Last but not least, the poorly named Management Service, which only really ‘manages’ transferring files to and from Amazon’s S3 cloud storage service. This service uses Amazon’s AWS SDK for .NET to take care of the data transfer, and is what allows you to migrate data to and from ShareFile’s storage, and the StorageZone.

There are a couple of configurable settings for this service as well; here they are with their default values:

  • httpMethod = https Transport method; secure or non-secure
  • HeartBeat-Interval = 5 Interval in minutes
  • Recovery-Interval = 3600 Interval in seconds

Well, I hope this post is useful for anyone who is using, or planning on using, ShareFile’s StorageZones feature. Feel free to share any other insights or thoughts in the comments!

Advertisements

ShareFile StorageZones Connector 2.0 Install Woes

I was recently tasked with implementing ShareFile Enterprise, and am executing on a design that entails the use of the StorageZones feature. In case you’re not familiar, StorageZones allows organizations to provide access to on-premise (private cloud) storage via ShareFile’s web portal, enterprise sync tool, the Citrix Receiver, and mobile access applications. In order to enable this feature, the ‘StorageZones Controller’ service (an ASP.NET web application) needed to be installed on an IIS7 server running .NET 4.5.

This sounds pretty simple, right? Wrong. The installation did not work out of the box, and I spent many more cycles than I should have troubleshooting it. In this post I want to explain how I got from start to finish with a seemingly simple process that became a complex ordeal due to lack of specific steps in the product’s documentation. Hopefully this post helps others running into this issue, which I hope I’m not the only one! 🙂

When I pulled up the installation instructions on Citrix eDocs for the StorageZones Controller 2.0 web service, I found them to be sparse on details. Here’s what’s currently published at http://support.citrix.com/proddocs/topic/sharefile-storagezones-20/sf-install-storagezones.html:

  1. Download and install the StorageZones Controller software:
    1. From the ShareFile download page at http://www.citrix.com/downloads/sharefile.html, log on and download the StorageZones Controller 2.0 installer.
      Note: Installing StorageZones Controller changes the Default Web Site on the server to the installation path of the controller.
    2. On the server where you want to install StorageZones Controller, run StorageCenter.msi. The ShareFile StorageZones Controller Setup wizard starts.
    3. Respond to the prompts and then click Finish. The StorageZones Controller console opens.

After following these ‘3 easy steps’, I quickly ran into several missing pre-requisites which required manual intervention; before being able to move past step 2, I had to:

  • Install Microsoft .NET Framework 4.5 (download link)
  • Add the Web Server (IIS7) Role Service

Following a reboot I was able to run the StorageZones Controller (SZC) installer, which required another reboot after it finished. After THAT reboot, the login page came up with a BIG RED error when I opened the SZC login page:

HTTP Error 500.19 – Internal Server Error

The requested page cannot be accessed because the related configuration data for the page is invalid.

Confounded, I turned to Google to hunt any known issues that might have been seen elsewhere, and couldn’t find any. There was a generic ASP.NET post on StackOverflow where someone found a mis-configured side-by-side, but I assumed mine was fine since it was a clean install. I then looked further into the following error details:

This configuration section cannot be used at this path. This happens when the section is locked at a parent level. Locking is either set by default (overrideModeDefault=”Deny”), or set explicitly by a location tag with overrideMode=”Deny” or the legacy allowOverride=”false”.

I tried playing with some .configs per other suggestions on MSDN (changing allowOverride=”false” to “true”), nothing of which yielded anything different from the 500.19 error. After getting nowhere fast for about 20-30 minutes I called support to see if they had seen this problem and/or knew what I was doing wrong.

The first number I dialed (800-4CITRIX) took triage almost 10 minutes to tell me that I needed to call another number (8004413453). I called the other number and was quickly connected to a customer service rep. However, the rep had no technical knowledge about the product I was installing, took down some details on the error message, and told me that an escalation resource would reach out soon.

With it already being late in the day, I decided to just move on to something else while I waited to hear back. The next day I was contacted by the escalation resource, and hopped on a GoToAssist for them to help me get to the console. They ensured me that we’d get it resolved, and proceeded to validate my installation, and do some basic break/fix tasks (re-install, reboot, etc.).

I started to become frustrated after what should have taken minutes quickly turned into many minutes, and eventually close to two hours of re-installing, rebooting (two times, every time), and various other poking and prodding. For example, after adding the ASP.NET role service, we started getting a totally different error message (404.17 Not Found), and started modifying .configs and adding/removing role services.

Near the end of the call (and the subsequent reason for the end of the call) the support representative insisted that the problem was being caused by installing the service with a user account other than localhost\administratorThis was after I already humored him and created, and installed with, a local administrator account (localhost\sharefile) because he stated that a Domain Admin account wouldn’t work even though it was part of the Local Administrators group, and wasn’t supported for this installation (which I eventually determined is not at all true). He also stated that ‘a lot of the steps aren’t documented’, which was beyond frustrating.

It was at that point I decided that I was getting nowhere even faster with support, and told him that I needed to end the call. After arguing that it would be fixed by simply installing with the localhost\administrator user account, I finally convinced him that I would figure it out offline since I wasn’t close to buying his unfounded assertion. After the call was over, I went back to eDocs to review the ‘System Requirements‘ section of the documentation and make sure I wasn’t missing something. Here’s what was listed for the web server pre-requisites:

  • Windows Server 2008 Standard/Datacenter R2, SP1
  • Install on a dedicated server or virtual machine. A high availability production environment requires a minimum of two servers with StorageZones installed.
  • Use a publicly-resolvable Internet hostname (not an IP address).
  • Enable the Web Server (IIS) role.
  • Install ASP.NET 4.5.
  • In the IIS Manager ISAPI and CGI Restrictions, verify that the ASP.NET 4.5 Restrictions value is Allow.
  • Enable SSL for communications with ShareFile.
  • If you are not using DMZ proxy servers, install a public SSL certificate on the IIS service.
  • Recommended as a best practice: Remove or disable the HTTP binding to the StorageZone controller.
  • Allow inbound TCP requests on port 443 through the Windows firewall.
  • Open port 80 on localhost (for the server health check).

The steps that I was stuck on were surely related to the items in this list that aren’t very specific. Take Install ASP.NET 4.5 for example; To someone that has never installed ASP.NET 4.5, this step is unspecific, and lacks any semblance of detail. While searching for clues on what was causing the 500.19 issue, I recalled seeing the following command to ‘Register’ ASP.NET 4.5 (4.0.30319) on this Stack Overflow thread:

%WINDIR%\Microsoft.NET\Framework\v4.0.30319\aspnet_regiis.exe -i

I decided to run this command and refresh the login page, at which point I got a new BIG RED error on what should have been the login page. This time it was a 404.2 Not Found error. Based on the error message, I started investigating the other pre-requisite that wasn’t very clear in terms of steps, and isn’t even relevant if the ASP.NET v4 extensions weren’t properly registered:

In the IIS Manager ISAPI and CGI Restrictions, verify that the ASP.NET 4.5 Restrictions value is Allow.

I found and opened the ISAPI and CGI Restrictions feature in the IIS management console, which can be found in the IIS section of the server-level node. I then found that while the ASP.NET v2 extensions were set to ‘Allow’, the v4 extensions were set to ‘Deny’. I set both 32-bit and 64-bit extensions to ‘Allow’, and was then able to get to the login page (great success!); whew..

And so, something that should have taken a couple of minutes ended up taking a couple of hours. Hopefully I saved somebody somewhere a headache (or a couple of hours) by doing the ShareFile product and support team a solid, and sharing clear steps that should have either been handled by the installer, or at least the technical writer who published this lackluster, detail lacking, setup guide.