Hi all, any advice would be much appreciated.
Our environment has about 2000 terminal server users in a variety of locations and we are investigating reports of slow response times for certain users and applications.
I have run some load tests on various servers accross our LAN (using AutoIT scripts) to test load on the servers and logged application response times. This has worked well and given justification for hardware upgrades.
However, we are also aware that some of the 'slowness' that users are describing is likely to be network related. - we are using Windows Server 2003 Terminal Server with out any other software such as Citrix or WAN optimizers.
Does anyone know of a way, or had any experience in finding a way that we can measure application response times from a user perspective over a RDP session on a WAN link? For example I would like to be able to log and measure the time it takes for a .pdf or image file to display and fully load on the client. The current AutoIT script logs how long it takes for the .pdf file to open, however, it doesn't represent any delay in the RDP client screen refreshing.
Ideally this would help us measure how many Terminal Server users can be accomodated at branch locations given a certain WAN connection, and justify the purchase of any additional software with a view to increase the ratio.
Any advice would be much appreciated.
cheers,Damien.
If I were a salesman I would order a Porsche and sell you Edsight and WANScaler
I would start by looking at the locations of the Servers compared to the data and then the users to the Servers. And break it down a bit. Also if you can deploy simple scripts to the users to get latency info via Ping etc.
Ask the Sysadmins to place some QoS on ICA and start limiting B/W to printers. Then see how it goes.
Also get an eval copy of EdgeSight (I think it is only avail for 30 days on a Temp License)
Let us know how you get on.
--Emil
That's funny Emil! You certainly responded like a Salesman (e.g. didn't pay attention to the fact that this is a pure TS environment!)
This is a tough situation. Pure TS in a geographically dispersed user community has serious drawbacks if your "farm" is centralized. To really be able to provide useful advice, one would need more info than what was submitted, but certainly bandwidth is going to be a major concern right up front.
Rather than run a script to simulate load, I would simply create identical performance logs on each server and run them for a week or so. This should provide the picture on server resources and network. If your WAN is Telco managed, you can ask for a usage report. If managed in-house, take a look at what tools are available on the appliance.
To the question if I know of a tool that can quantify and record latency of user interaction - I don't. Business decision-makers have to trust what their employees are saying to IS. If need be, get off their duff's and make a trip out in the trenches and see what's happening in the real world.
Assuming that your servers are up to the task, you may have to think out of the box in terms of how your WAN is implemented. For instance: Rather than dedicated or fractional T1 service, purchase commercial cable service and employ VPN appliances at each remote location & go from 1.5Mbps to 5Mbps. You can de-centralize the TS servers and that could improve the user experience. Of course, that has consequences as well.
Samuel A. RodriguezSr. Systems Administrator
Whilst Telco line usage reports might show you longer periods of over-utilisation, they often won't actually give you the detail that you really need.
A typical problem is that some sort of "big data" is sent over the WAN link, which will tend to push all interactive traffic out of the way (and THAT is when the users notice it!). Often this may be for only 30 seconds..... and 30 seconds is a long time (go on, look at your watch, and wait for 30 seconds....).
The sort of traffic I'm thinking of here could be a print job, an email (with an attachment), downloading a file, getting a video or audio file.... these may, or may not, be business related. What sort of "other traffic" do you have (or do you THINK you have) on your link?
Identifying the traffic can be a pig (something like a packetshaper can be really useful to do this). But once identified, it often needs only some simple methods (eg router QoS) to fix it ..... assuming that you do, in fact, have a big enough pipe to start with.
So then you start looking at compression boxes, like Wanscaler, Expand etc. All will claim to improve your throughput, but to be honest, none will GREATLY improve your Citrix ICA throughput (this is already compressed: yes, you can disable the compression, and use an external compression box.... but I wouldn't expect the gains from it to be THAT big), but the *other* traffic may well be quite compressible. So maybe worth a look at.
Or maybe it is time to get a bigger pipe....
Hi all,
Thanks for your insightful replies, they have given me a lot to think about, and confirmed many of our assumptions - ie. that RDP / Terminal Services is inherantly a bursty application / protocol.
After doing some user analysis we have found certain websites (flashearth.com) and user actions (scanning, printing) greatly impact on the RDP performance and WAN utilization.
Still, I would like to be able to provide hard numbers that the business can understand in terms of utilization / user response times. I've looked at using AutoIT scripts installed on the client end (XP PC) running a RDP session to the TS and then monitoring pixels on the client to determine when an application has loaded and refreshed from the client perspective.
I'm not 100% sure that this is the way to go forward on this however, and what real benfit it will provide after the assumtion / acceptance of RDP being bursty.
In answer to a couple of questions, the only traffic that shuold be going over the link is RDP, however, we have noticed Tech Support transferring ISOs over the link, and no suprise that users have noticed a dip.
A way that we plan to go forward it to provide network bandwidth visibility (Netflow) to staff at the remote site. This way when users complain about response times, then IT staff at the remote site can look at the graph, and try to determine what actions have thrashed out the links.
One step forward, two back it seems!
cheers,tofe.
(Reminder to self: read the user's question carfully before replying!)
You are using Terminal Services and RDP, not ICA. So, a few thoughts:
- ICA uses noticibly less bandwidth than RDP.... to use that, you'd have to install presentation server etc, but it could be cheaper than paying for more bandwidth :-)
- Whilst edgesight is designed for use with ICA, it may actually work ok with RDP, so would give you a way to monitor the end-to-end perfomance (or lack of)
- Printing is a PITA as it can use HUGE amounts of bandwidth. I wonder if it's worth looking at something like Uniprint, which can drastically reduce the bandwidth used for printing.
- Implement QoS over the link: this could be simply done with your routers (if they support it, or for a more "complete" solution, look at Packetshaper.
- Compression products (WanScaler, Expand, Packetshaper) may be effective, especially in reducing the bandwidth used by printing scanning etc, thus leaving more space for the RDP traffic.
paul
@Sam - Ouch!
I normally run applications/desktops with reduced resolution and colours to see if it is BW related, also around 200+ms (rondtrip) threshold where RDP will start to deteriorate and as always if you have packet loss this will also be a factor.
Out of interest, what sort of size links do you have? How many users on such a link. What technology is the link (point-to-point? MPLS? VPN on Internet?). Just thinking about REAL bandwidth availability, and packet losses.
People often grossly underestimate the bandwidth needed to run RDP: I was doing some simple testing yesterday, and had "the matrix" screen saver running (rather graphical)... it was using just over 1 MBPS :-)
I think one problem is that the "standard figures" that we still used are rather out of date: many applications are a LOT more graphical than they used to be (hello, Mr Paperclip! :-) )
(Have you seen the episode of "Air Crash Investigation: as a result of a plane crash, the airlines now use a higher "weight per person" (by about 30 lbs) than they used to, as passengers are on average, much fatter than they used. They also found that carry-on luggage is a lot heavier than it used to be. Kinda like our modern windows apps!)
Paul
@EmilBeck - friendly jab. You don't need Edgesight to figure this one out.
From the client perspective, it might appear to be latency, when in fact it's contention. Keep it simple. Make a decision. Resolve the problem. Move on.
I know these days it's all about several meetings a week (on status of the same 'ol problems that are taking forever to get resolved) & one certainly want's to show their abilities in Powerpoint and Excel. But I ask myself: Why do administrators care to complicate matters? Why do the business folks need 1000 rows of data on a pivot table to figure-out that there is insufficient bandwidth?
RDP and Terminal Services are not bursty. User workflows are bursty. You don't notice it on the LAN because the user community is segmented over 100Mbps switches and probably have a 1Gbps backbone to the servers. Unfortunately, WAN technologies haven't advanced that much and we still find ourselves with 512K - 1.5Mbps (T1) links to remote offices (Frame Relay/Fractional T1 etc). Most of us have that much bandwidth for just ourselves on our home DSL line!
All you need is one individual to send a print job of 10 pages of powerpoint and 4 copies of it (for a meeting they're going to have in 5 minutes) to flood that link for 15 minutes. Copying an ISO has the same effect. I'd say, get a basic WAN utilization report for that circuit and start there.