My scan results in crashes? What causes this?

[clicker]

I've been running long eValid scans on a complex website and sometimes I get a complete failure...you know, the "send this message to Microsoft" type of failure.

What can be causing this? It is not every time and every site. But it is a real pain so I really want to know what the issue is...

I'll wait for a response here...

Mr. Clicker

[eValid]

I've been running long eValid scans on a complex website and sometimes I get a complete failure...you know, the "send this message to Microsoft" type of failure.

What can be causing this? It is not every time and every site. But it is a real pain so I really want to know what the issue is...

I'll wait for a response here...

Mr. Clicker

Well, hard to say. So if you can send us the actual page on which the crash happened we might be able to debug it.

But maybe this little piece of technical detail may help out here: eValid is built using the DLLs from the IE browser that is installed on your machine. The eValid product this is a "clone" (for lack of a better term) of the IE browser.

The good news that this means eValid processes and renders pages in the identical same way as the IE browser does -- this is important to understand because it assures the accuracy of eValid-product timing and size data. But with every bit of good news there is, you guessed it, some bad news.

The bad news is that eValid also includes every IE feature, including some that do cause the browser (IE or eValid, no matter which) to fall over in some strange cases. It could be that THAT is what you're seeing.

We have heard of this before, on very long scans and/or on scans where the application is particularly difficult. Sometimes it helps to have the latest IE
installed on your machine (IE 8 is out now, at least in Beta)...

To diagnose the problem -- to repeat -- we need the page where you see the product failing in order to be able to see if it can be fixed.

** eValid Support

[efmiller]

[Background: We have been working with this customer to try to understand why the very large site seems to cause problems. The site is so challenging that we here are all trying to see what the problme may be. We moved the conversation to the forum because of the general interest we suspect users have in this general area of concern.]

We have been trying the run on different machines and we can report that the crashes appear to go away on a Vista box running IE7.

However, there are other problems.

To get the run to conclude, we limited the link count to 100,000, but when the scan got to 9,024 pages (it took 9:51 hours to do this) the eValid application became "not responding".

We checked and found that the 3 GB memory had just less than 40 MB available -- less than 1%. So we think that the machine is so busy (that's why you see the "not responding" message) that it can't complete.

That site uses VERY long URLs...it looks like the are over 100 bytes each and some appear to be even 200 bytes. The internal table that eValid uses needs ~2X the total URL string length, so a 100K link table with 100 bytes per link implies an in-memory requirement of about 20 MBytes for just that one table.

Our guess is that this is a performance limit on sites that have very long URLs, that is, that the total size of the internal table probably has to be less than 10 MB.

We'll post further results here as we find them. But the recommendation is,
try the scan on a machine with larger RAM (which may have to be a 64-bit machine because of the 3 GB limit on the 32-bit architecture in Windows).

-E.F.Miller

[efmiller]

Continuing on this thread, we are going to try a run with the link count limit set to 75,000 to see if this will get all the way to the end.

Also, we looked at some of the lengths of the URLs and we found (without actually calculating, just estimating) that the earlier guess on the URL lengths was low: some of the URLs are in the 400-600 byte range.

So, this may be another factor contributing to consuming the RAM.

-EFMiller

[efmiller]

Further to the prior dicusssion...

It looks like a 3 GB machine's limits are about 50,000 links, which is what this scan did. That is for this particular website, where the URLs are very very long, e.g. 500 bytes or even much longer.

We tried higher link count limits and in several runs always ran out of memory and the machine came to a halt.

The problem is the length of the URLs.

We tried combinations of all cache, no cache; same issue.

When you have so many links, and they are so long, the table internal to eValid gets so big that the machine is spending 99+% of its time swapping to "update reports".

The machine OS didn't fail and eValid didn't fail...it was at 100%, no available RAM, but still trying hard.

Sorry, guess we just need a bigger machine...

[eValid]

Summary...

After careful analysis of this site we recommend:

(1) More RAM on your machine.

(2) Tighter scan constraints so that you don't hit machine
resource limits.

(3) Partition of your scan into sections on the target website.

The eValid Team