November 27, 2007

New beta version of Burp-Suite !

Today I saw an interesting blog post from PortSwigger , finaly announcing new beta release of Burp-Suite . Reade details here . I`ve not got the chance to fully review new beta release but some known bugs I was aware of , are hopefully fixed . And Burp-suit now have it`s own integrated cookie & session analyzing tool , named as "Burp Sequencer" . "Burp Decoder" is another addition . Although a encode/decode tool was really missing in suite but I still prefer using IO-Active`s Morf for this purpose , among the Decoder .
There are many web-application auditing packages and suites out there. Skipping Burp , here is list of some of well-designed and usable ones :
Based on nature of audited application , and depending on our plan on how to audit the web-application , #1 suite changes usually . For example , if you`re going to fuzz parameters SPI`s fuzzer is second to none . If you look for automated traffic mannipulation and data replacement while relaying traffic , Burp-Proxy is the best answer , and so on . Although we usually have all or most of these tools available in our tool-box , but there`s usually a suite that`s used more occasionally than others , and I`ve to say for me , Burp-Suite is the one !
There are two exceptions in mentioned list . SPI-Toolkit is a commercial suite, and SPIKE-Proxy is actually a web-application fuzzer more than auditing suite , although it have powerful features for manipulating traffic while relaying through it`s web-interface .
Time to leave office . till next post...

November 22, 2007

Embedded fun - Hacking MP3 players (MuVo-v200)

As I promised , here`s first post on hacking embedded systems.
In a very brief sentence , embedded systems are family of devices , designed for very specific tasks. read Wikipedia article on embedded systems if you`re not familiar with them.
Every single embedded system (or device, if you like this word more) you may find around , have it`s very specific operating system , which is designed based on a core component ,usually a SoC (System On Chip) . I`ll call it 'chip' from now on . Depending on functionality of device , different chips are used , and depending on used chips different operating systems are loaded and executed . In case of embedded systems , operating system is called Firmware .
It may look funny but from your MP3 player or digital camera to your micro-wave oven , all have their specific OS and are considered an embedded system . Basically most of those digital devices around you are some kind of embedded system . You got the point ,right ? ok let`s continue .
Focusing on multimedia players , it`s easier to break them apart and study every single piece of an embedded system. Everybody out there have some story about hacking complex embedded devices such as wireless routers or IP cameras , and everybody try them because it`s actually some kind of linux hacking and programming challenges IMO . There are many creative and cool hacks out there for IP cameras or printers , turning them to your attack platform for further internal compromises. Hacking IP cameras is really fun and I`ll dedicate another post explaining how to convert an AXIS ( or any other brand ) camera to a proxy , for browsing and attacking internal networks.
Back to mp3 players, I consider them very popular devices ,every one carrying one in his pocket or bag . So finding any possibility and vector on them to turn them into evil devices would affect very high number of people. Enough theory , let`s get our hands dirty a bit .

So I decided to know my Creative MuVo-v200 better. Although I learned how to do it the lame way , but I`ll tell you how not to do it , without voiding guarantee of your device. First attempt to hack my MuVo was just to learn internals of an embedded system , but the more I learned about them , better ideas appeared on abuse possibilities .

First step was checking used chips and hardware parts . well , an mp3 player is nothing but a mix of a flash-memory , LCD , and a DSP ( Digital Signal Processor ) . DSP is heart of every mp3 player and can be considered the SoC I`ve mentioned before . DSP is the part boot and use firmware of your device from it`s internal memory . Note that flash-memory used for storing firmware is apart from the flash-memory used for storing songs or files. In case of MuVo-v200 , file storage flash-memory can be unplugged and used as a normal flash disk .
As I didn`t knew I can use Google to find used chip on my device, I opened the cover to take a look :

Heart of device is a SIGMATEL STMP3510 DSP chip .
As usual , checking vendor site returned nothing interesting but few technical marketing papers. There I just learned few tips about internals of STMP 35xx chips. Not bad , but far away from 'enough' . So I googled for more technical details about chip and found interesting notes here . Now I know that my MuVo-v200 is using Motorolla DSP56004-compatible 24-bit signal processor as it`s actual heard of SoC . This also tells me that I should look for technical notes on developing code based on Motorola chips . This piece of information will be used later on studying firmware . Searching for more information about used Motorola chip landed me here . Not every line of this document was interesting for now , but at least gave us the point on what other documents we should look for , for farther technical details. These two data sheets ( 1 , 2 ) also had interesting information inside but second one was really boring and actually had nothing to do with what I was looking for. Finding information about your chip some times may not be this simple, and usually some documents are provided only to resellers or manufactures. Here`s the time to try p2p file-sharing networks . Reading and reviewing these hardware specifications is really boring so that`s enough for now . Now we have enough information about capabilities of the chip and the way it really works. Poor firmware developers, they should study them line by line .... :-/

Time to check firmware ,and software part of the mp3 player. At first glance you may think where the hell from , I can find the firmware of my device ? Well , there are some known ways but we all prefer the most lame tricks :)
The hard way is using some USB device debuggers , which are consisted of a hardware device ,and some software parts using the hardware as their source . We can use them to gain direct access to memory of chip , and dump it . hmm not so interesting for somebody like me know any shit about this topic. so let`s find easier method.
Checking vendor web-site, I noticed that they provide updates for firmwares . cool ! In general you should always go check "support" or "downloads" section of your vendor`s site . If they release anything related to our mission , it`s usually placed there . Files related to my mp3-player are published here .
As you see , among new firmware package there`s another package called "Mass storage Driver" . We get them both . later I`ll tell you more about second package . Now we have a setup package for upgrading firmware . I installed it and instantly checked packages contents . An executable file responsible for handling GUI works, and two other executable files looking like to make the device ready for upgrade process . Other interesting contents of package was two device-driver files , and checking related .ini files beside them told me they are used for loading/unloading device in repair-mode . Before attempting to really upgrade firmware , don`t forget to backup files stored on memory of your devices . Upgrade will usually erase them (although not necessary) . Second , try to find current version of your fimware from vendor site or any other trusted place , and have it for unexpected situations . It`s because some times new firmwares just fuck your device and REMOVE some functionalities you had before , and if you miss old firmware version , you`ll loose them for ever . In case of MuVo , new firmware reduced power of output sound level ! Although they did it to take care of health of my ears but I really missed old loud sound . If you don`t know current version of firmware, firmware upgrade program usually tells you , before upgrading to new firmware .

When you attach USB device to system , OS detect it as a simple storage media , let you upload your files to device . But as you`ve already guessed , this way we can not access memory-chip to upload/download firmware files . Firmware upgrade program actually look for attached device , and try to mount it in a specific way , to be able to access ( read/write) low-level data through USB interface . After successfully mounting device in raw mode , it will attempt to upload new firmware files .
MP3-players firmwares are usually consisted of four binary files. Although files I`ll mention are specific for MuVo players , but usually any device which is build based on same chip (SIGMATEL in our case ) have same or alike names and have exactly the same functionality . The reason is that , although different vendors ship different options and packages , but if they use same chip , they must have follow same specific rules provided by chip manufacture. Let`s review firmware files .
Monitoring firmware upgrade software with simple tools like regmon.exe and procmon.exe (SysInternal tools) you can easily monitor activity of software and upgrade process. Watching upgrade process with these tools , it`s possible to see names and location of files software use as firmware and upload them . Although firmware binary files are really obvious and can be easily differentiate from other files in installer package , but I just mentioned this way in case they are not easily located in your environment or package .

As you see , after some junk events , software finally begin reading small blocks of data from firmware files , and write them to chip memory in raw mode . So now we have name of pieces of firmware files and we can turn back to contents of setup package . There are four files , and I`ll briefly explain their purposes . is self explaining . This piece of firmware acts like a boot manager in your pc , for mp3 player . When device is powered on , this binary is loaded in memory and try to load the right next-stage of firmware in memory of device. The point is that these devices usually have two working modes . One working mode is when you attach device to your system as a removable media, and the other mode is when you power-on device by battery and listen to music ... . Device have two different firmwares for above modes , so Bootmanager check and see which working mode it`s in , and load appropriate firmware. is the firmware bootmanager load , if it detect device is connected to a USB port . This is actually the firmware introduce your mp3-player as a removable storage media , when you plug it in your pc . is the firmware loaded by bootmanage , when you normally use your player . this is the part we should plan for evil modifications .
resource.bin is a binary file containing resources as it names tell us . When you use your mp3 player you`re working with some menus, you see some images ,texts and etc ... . This is the file holding them all . I`ll explain how to have some fun with this file .

Here you can find more information related to MuVo and other players based on STMP chip.

So far , we learned how to find information about hardware part of our device, check used chip , find it`s capabilities and finally grab firmware files and do a sample upgrade. what next ?
Easy task is to hack visual effects of your mp3-player and customize it to amaze your friends by some odd contents on LCD of your mp3-player. In order to do this , you don`t need to know much details about firmware itself . All you have to do is modifying right contents on resource.bin file , and upload new firmware to your device.
Of course , modifying resource.bin is not as simple as opening it in your favorite editor and change binary data . Firmwares and linked files are usually packed and encoded (some times even worse , encrypted ! ) so modifying raw binary data will corrupt the binary image . In case of SIGMATEL , the hard work of identifying used mechanism to decode firmware and resource files is already done , and to hack your resource.bin file , you simply need to download and use a tiny program . As an example , you can modify and replace resources "0651" and "0652" to replace Creative logo , with your own image . So when you power-on your player you`ll see modified image .

There`s only one note to remember before attempting to resource-hack your mp3-player :
If upgrade process terminated before successful finish by ANY reason (system crash in my case!) be warned that your mp3-player will NOT be usable after failed upgrade process . Not as mp3-player nor even simple flash-disk . But don`t panic , it`s easy to recover and clean-up messes . Remember "Mass Storage Driver" I mentioned before ? Here`s where you need it . All standard mp3-players have a recovery mode for such times. By using a special set of instructions ( like pressing and holding a key on player) while plugging it to USB port , you can switch to recovery mode . In case of MuVo , it`s holding and pressing 'play' button while plugging it to usb port , and you should hold the key until system detect device . Now you can run firmware upgrade software again , and upload firmware again . This instruction may be different for other manufactures , so ask google .
Here the easy part finishes .

Hacking MP3-Player firmwares:

Let`s discuss some serious works and review possible real attack vectors . This is where things get really interesting . Abusing portable media/devices as attack vectors for infecting and attacking end-users is not a new idea nor a new unknown techniques. I`ll briefly review finished works on this topic and finally explain my idea .
Portable devices , or to be more specific flash memories , are well-known for being abused to attack users. A simple and easy vector is infecting file contents and using the Autorun , to execute our malware . Since availability of U3 flash memories, things changed a lot , and new vectors discovered and implemented . But they all were using same trick : using Autorun to execute a file , when plugged to system by user . My favorite story about this trick was the way some attackers tried spreading some infected flash disks among corporate users , and almost all of them curiously plugged the infected flash disk ,and game was over .
The other known attack vector based on using removable devices (This time focusing on a more general vector , USB interface ) was hacking the flash chip itself, to cause overflows on guest operating system . This is also discussed publicly and some demos also presented , in BlackHat 2005 . I contacted Barral recently and he was nice enough to provide some detailed information about their previous work and findings . A detailed post is required to cover findings in this field and some more technical details, which are not presented anywhere else yet . As this vector require extensive work on developing code in kernel land , and I`m a lame at that , I`ve almost paused research on this vector , till get another chance to work on this with other researchers like Barrel ( Thanks for your help Darrin , if you read here! ) .

Back to my own idea , I`ve not been looking for yet another way to autorun malwares placed on flash-disk , nor exploit any flaw through USB interface or device-drivers, as Darrin and Dave presented in BlackHat .
Recently I came across an interesting article titled "Attacking consumer embedded devices" in INSECURE MAGAZINE 14 , covering this topic (hacking embedded devices) . I highly recommend you read it , if you`re not familiar with embedded device hacking techniques .

The new idea is hacking (customizing) firmware of mp3 players in some way that they act as our infection engine. Looking at current available SDK packages for developing mp3-player firmwares , confirm this possibility . New mp3-players ( like MuVo-v200 ) are now capable of writing to flash , among various reading capabilities . And implementing _writing_ to files means much ! Current firmwares are capable of doing complex encode/decode of music files , or browse and modify (currently only delete ) files on attached flash memory . So why not add new module to their firmware to do our tasks , say modifying stored files on memory ?
An output of this research will be a customized version of firmware , which is capable of doing following tasks :
  • Scan contents of flash-memory , locating interested file formats
  • Based on detected file-format , select a pre-defined payload and inject it into stored files
  • Modify flash-memory content and inject (write) new autorun files to it . (Smart implementation of old hack!)
  • Load appropriate modified , to trigger classic overflow attack vectors
  • Some rootkit functionality! , to steal interesting files and hide .
  • {Many other interesting options...}
In order to achieve any of above tasks , extensive work is required for two main reasons. First , all works should be done under ASM , and even more painful not normal ASM , but a limited version due to architecture of used processor which is NOT a general purpose processor . Second problem is learning how to implement functionalities which require learning current codes through reading them , a tedious and hard work even for ASM experts !
In order to be able to write assembly code for used chip , developer must first locate a reference . As these chips are designed to be re-packaged , hopefully it`s easy to find such reference manual . I could find assembler reference for used Motorola DSP chips here . Such references can not be used to learn assembly from base , but should be used by an experienced assembly developer , to migrate his knowledge to new platform .
Next step would be finding a SDK , providing a base for developing new firmware . Don`t even think about developing a firmware from scratch , because developing new firmware based on available SDK is complex enough ! SKD packages are usually commercial and are not available to public . Creative for example , in order to develop his custom firmware for branded mp3-players should purchase SDK specifically designed for Sigmatel chips , from Sigmatel . And prices are not usually much end-user friendly ! But p2p file-sharing networks are not died yet ;) I could find and download an old version of Sigmatel player SDK easily . it shouldn't`t be hard to locate and download it for other manufactures.
If you remember first paragraphs of this post , I mentioned that there are some documents related to technical details of DSP chip which are not provided free and are classified in some way . Hopefully these documents ( like STMP35xx users manual) are provided with SDK . The SDK package I downloaded had these documents among some other useful PDF files .
So far , based on gathered information , we can build our own mp3-player and customize and develop a custom firmware for it .
Next step , would be building new firmware from source code . If we modify provided codes from SDK or simply want to test raw firmware available in SDK , we should first built it . As we`re cross-compiling , we should find and use a custom compiler supporting target platform and architecture . Asking Google again , reveals that an old customized version of GCC compiler ( 1.x version ) is usually used to compile and build the firmwares based on Motorola 56xxx chips. Here is where I got above note from , but it helped me to find many other resources related to compiling firmwares for Motorola 56xxx chips ,and finally here`s the compiler we need for our work , DSP56k-gcc .Turning back to functionalities of customized firmware , I`ll review some of them with more details .

Scanning contents of flash-memory for interesting file formats , let us locate files in root and sub-directories and reserve their path and extension for later tasks. Some interesting file formats to look for are exe, linux PE files , dll , doc , shell scripts (bat,cmd,sh, etc...) and any file that let us inject payload inside them . The list can be longer , covering media files like mp3,mov,... and possibly ANY file-format that there`s a known vulnerability exist for it !

Injecting payloads based on detected file formats is the main goal of whole of these tasks . A quick and easy implementation in customized firmware may try to infect .exe files as they are easy to modify and it`s easy to code it in asm . You may think that who`ll store .exe in his mp3-player ? The truth is that nowadays mp3-players and flash-disks are actually mixed together and most of people use memory of their mp3-player as their portable stirage media . So many popular file formats may be stored there , among music files . Skipping this possibility ,there are still many known vulnerabilities affecting multimedia file formats ! you can find some interesting samples targeting mp3 , mov and other popular formats in blog . Skipping the file-format story and focusing more on injected payload , it can be virtually ANYTHING . the only limitation would be storage of multiple payloads in limited space available for storing firmware files . Payload can be a self-spreading trojan , a key-logger , or anything you imagine .
Infecting file formats is really devilish because like other classical methods , attacker will not make any noise on target system . Classic methods can be blocked with simple prevention methods like disabling autorun . Attacking overflow vulnerabilities are also very unstable , and drawback on using this method is that an end-user can not actually be victim of such attack . because a modified flash-disk will no longer be able to serve it`s legitimate usage if it`s designed to attack device-drivers . Finally it`s not easy to build such flash-disk to be able to attack many different targets(and platforms) . In order to target different systems you must re-program your evil flash-disk .
But using discussed technique , end-user will not notice anything wrong , or even suspicions . Do you really check size of mp3 files every time you listen them? I bet you don`t !

Injecting Autorun files can be used if injection is not possible , for example when there`s no file stored in flash-memory . or when there`s no victim file-format available . this functionality can be used to write autorun files , so when flash-disk is attached , defined tasks/commands are executed .

Load appropriate modified , will let us use classic overflow attack in more stable and reliable way . First good point about using multiple files , is that we don`t have to re-program our flash-disk for every different target . We can implement a module inside firmware to automatically load best second-stage . Even if we can`t implement an auto-detect method for a smart attack , we can still manually select the right stage from mp3-player menu !

Since this technique (customized firmware) is stealth from user point of view , it will last much longer than classic attacks based on flash-disks . Even a sharp expert will have trouble back-tracing infected files back to firmware of an employee`s mp3-player ! Most people will quickly format the flash-disk you may have gifted them anonymously , but how many of them may really upgrade firmware of device ?

I`m still working on this topic very slowly ,and there are some parts that I can not handle alone. So if you`re interested in this topic feel free to contact me . I may present findings and results of research among some demos in an official upcoming con. , of course if I find a way to bypass my soldier-work and get out of country !!!

hmmm this post is a bit longer than usual , isn`t ?

/* I may update contents of this post to keep it updated or fix errors , or add new items */

November 21, 2007

Women and Security (Nmap case)

While browsing Nmap changelog, I came across this paragraph which looked very cool to me:

Reworded an error message after a woman reported that it was "highly offensive and sexist". She also noted that "times have changed and many women now use your software" and "a sexist remark like the one above should have no place in software." The message was: "TCP/IP fingerprinting (for OS scan) requires root privileges. Sorry, dude.". I checked svn blame to call out the insensitive, chauvinistic jerk who wrote that message, but it was me :).

Things change, and women too ! :-)

Btw , today Nmap is trying to be confusing . I tried to do a normap ping scan against some internal firewalled hosts ,and nmap was responding me like an idolt ! normal ICMP through 'ping' command works fine , but Nmap failed to find any live host .
Trying different ping scan methods and switches didn`t help so I became more suspicious. running favorite sniffer told me that Nmap is not sending any ICMP packet at all !
Here`s what Nmap is sendinf as ping proble :

Noticed highlighted protocol type (in red) ? Legitimate flag should be 01 ( ICMP ). What`s going on ? Am I missing something ?
I`m suspected to used platform (win2003 SP2) . I`ll check it from another host and update post if anything new appeared .

November 19, 2007

An overview of using zero-day exploits in penetration tests

This is yet another post focused on penetration tests and misleading points about them .

I`m hearing a lot about zero-days , un-patched bugs , in the wild exploits for just reported vulnerabilities and many alike terms that finally will lead to a painful compromise , in case the target is not someone with "Defense in depth strategy" in mind. At the other hand I`m faced with people, thinking about professional penetration testers and skilled auditors as magicians !

Talking about pen-tests , many people think that pro. auditors are successful at their tasks because they always have some 0-day kind of bug/exploit/technique/trick available in their hands. It`s really common to hear that "He has probably used a 0-day to own them..." , "You`re ling me , I checked for every single possibility and could not find anything useful. Do you have any 0-day exploit for service X ?!! " , and so on ...
Yes , zero-day stuff will some times guarantee success of missions , but are they really used every time ? Will pen-test fail if every single host/software is updated to latest version ? Will you be hacker-proof if you monitor everything and keep your softwares and systems immune against zero-hour threads ?
I`ll instantly answer NO , to above questions. Doing pen-test for at least 4 years and successfully compromising many banks , government & military services , mission-critical infrastructures and tens of individual companies , I have to say that success of mission was very rarely depended on using private or zero-day stuff ! I`m not saying 'never depended' because some times customers specifically ask for being tested against unknown and zero day vulnerabilities and techniques , but that`s not a general case .
I divide customers into two groups . Those who understand the risk of zero-day attacks and willing to pay enough so their pen-test (and R&D behind tests) cover it, and those who blindly tell you "Try every possible way to break into network " and don`t care about unknown vulnerabilities , assuming being safe against attacks on latest-versions is enough .
So how tests lead to successful compromise , if we`re faced with a patched infrastructure and we have not used ANY zero-day exploit (Although we may have some) ?
Reviewing past projects results, and watching the way most teams manage their networks will tell us that there are always some design flaws and management problems , guarantee the success of hackers .
When I mention design-flaw , I usually mean the way networks are planned and designed , or the way small pieces of a big infrastructure are putted together . This is really common in big networks. Telecommunication or banks networks for example, have many entry/access points among many back-end systems . Assume a bank , have X number of services available to customers , and customers can access these services in Y different ways , and there are Z number of known attacks/vulnerabilities available for every combination .
Therefore we have X*Y*Z number of ways to gain access to interesting data ,and we`re always allowed to test at least few of them. Insecurely allowing branches to connect to central office , is an example for this group of vectors , causing serious risks that affecting infrastructure .
I mentioned management problems too . And I mean the way networks are managed . This covers they way a password-policy is deployed ,to the way administrators keep their systems up-to-date ,configure services and devices ,and monitor & take care of human-factor mistakes . Even a single insecurely configured service can put whole of linked systems at risks , or a single missed security patch can be used as entry point for digging other systems & services .
If you search enough and investigate everything carefully you`ll always find an out-dated thing ! Yes , out-dated means something that have some KNOWN vulnerabilities . This thing can be a firmware , a third-party service or software , an active-x , or even old version of a protocol .
And finally about weak/default/missing passwords ?
There`s yet another interesting vector to investigate , which is not taken seriously by many people trying to break into a network . Foot printing .
I wrote multiple paragraphs right above this sentence about finding X and Y , checking this and that , looking for design flaws and etc ... . Have you ever imagined where all of those information may come from ? They all depend on a good foot-printing process which is the very first of every penetration test . The more you focus on it and do it carefully , the more findings you`ll have to work on for next steps.
I personally had some experiences that effective and deeply foot-printing a target , resulted in finishing pen-test at first step ! Yes , believe me or not , if you search enough and carefully there`s chance to gain access to some critical knowledge about target , without touching a single host in target`s network which may result in instant compromise of data/systems/users . Even if this case do not apply , you`ll always learn very much about your target .

So what`s the point behind writing all these ?
The point is that , why should you use zero-day stuff at all , when there are MANY ways to get close to target without using it ? If a hacker/auditor correctly try every mentioned vector , I can guarantee that he`ll be inside and intruded enough BEFORE reaching the end of first loop of try & discovery !
Some people may ask "why should we waste that much of time & energy, when there`s chance of almost instantly 0wning final targets by using few zero-day stuff ? "
Nice question . The answer is that zero-day stuff are not provided freely or even easily and even more important , not only they are not free but they are usually very expensive ! So the point is that zero-days have their own value and cost and can`t/should n`t be used like normal stuff unless it`s really necessary or asked by customer. Value of every single zero-day thing , should make you think that does it really worth using it against target ? Let me be more clear in providing an example here . Will you use a zero-day vulnerability that costs 20.000$ against a 10.000$ target ? It`s highly depended on you but most people will answer no . Of course you may answer like you use this 20k $ vuln 5 times and you`ll be the winner . This way I`ll wait for you to come back later time , willing to pay 50k $ for same flaw because you have wasted and burnt it previously for multiple cheap targets , and now you`re faced with a hardly locked down target which costs 200k $ ! got the point ?
There`s usually so much time,energy,knowledge and experience behind every single zero-day you gain access to (Either by finding it yourself or paying for it) . So they should be used like golden and final bullets , when there`s no way not to use them .

So , next time you failed in your intrusion attempt , it`s better to look back and see how you`ve finished previous steps and how much careful you`ve been in reviewing and working on your findings , rather than asking God a zero-day . Did you really tried every possible way ? I bet no !

There is still much to write about zero-days , like why/when we should look for them , or the idea behind current zero-day markets and clubs . I`ll prefer to leave these topics for another post. I`d appreciate your comments telling me how to continue this topic or even completely stopping it :)

November 14, 2007

Web-applicatoin Vulnerability Scanners War

Do you remember Larry Suto`s paper on comparing few commercial web-app scanners ?
Suto evaluated actually three scanners (NTOSpider,AppScan,WebInspect) , by targeting 3 web-apps while monitoring scan process with Fortify Tracer product .
Well , a paper full of big names !
At first glance , paper looks interesting to anyone unfamiliar with web-app testing , or new to automated tools . I was also surprised how one of evaluated products (NTOSpider) remained anonymous before release of the paper . believe me or not , MANY people changed their minds and refreshed their ratings based on published paper . I think including "code coverage" among "Fortify Tracer" in paper , fooled many readers .
After releasing paper , I was expecting evaluated companies to respond to it , as paper made a lot of noise in community , but we saw a strange silence since release date , till today . As the winner (based on paper ratings), NTOobjective was just busy responding to hundreds of trial-request emails and calls ,and also happy about being rated #1 ! Why shoud they care about accuracy of published paper , when they`ve got their free marketing chance ?
About WatchFire (IBM) and SPI (HP) two companies being supported by world leading researchers in their teams and also unlimited budget for their work , a technical response was expected soon or later . SPI was the first ,to present their response .

Today I noticed SPI released the paper , revealing some realities about the process of evaluations , and how a real evaluation report should be . At first glance you may assume it being a paper to save SPI from being the bad looser in test , but if you finish reading it you`ll notice how clear and more-accurate real results are. And to save your time Jeff Forristal is trying to tell you "WebInspect is not really as bad as mentioned in previous paper!" and also "Suto`s methodology for evaluation was not effective and accurate."

Using WebInspect for so long time as part of my tool-box for (automated) assessments , I was curious about this new scanners war , and the way competitors are going to prove themselves the better one. I`ve to mention that I`ve used all of named scanners in real assessments and personal hard-core evaluations , among some other scanners not mentioned in this war . So I know very well what I`m talking about here .
Focused on technology and logic behind scans and of course accuracy , I`d instantly filter out only WebInspect and NTOspider , leaving other commercial products. Talking about AppScan I`ve to say that it really disappointed me every time I tried to get close to it and evaluating new versions/features released by vendor .
The day I began using WebInspect (back in 2004), most of automated (web-app) scanners were nothing but polished crawlers with capabilities to manipulate/inject some parameters in HTTP protocol and audited application , armored with number of hard-coded vulnerability checks related to known web applications , web servers and few application servers. There was not much intelligences nor effective methodology in background , and you had tens of false positives and missed critical vulnerabilities per scan . WebInspect and NTOspider were a little bit better and faster than other lame scanners and had few optimizations and customization capabilities inside , making them better than others . That was the reason I turned to SPI .
Other interesting point about WebInspect was SPI`s focus and continuous work on enhancing scanner engine and making scan modules more intelligent , rather than releasing burst of new vulnerability checks and hard-coded scan signatures . NTO was almost like SPI , but never as active and productive as SPI . So while closely monitoring NTOspider development and releases , I kept using WebInspect . ( feel free to call me a fan of SPI and their research team! )

Nowadays , scanners changed a lot . They no more try to relay on a massive database of known vulnerabilities and try to check them against target . They`re being designed to crawl and parse web application parameters smartly and check for common vulnerabilities by AI and few hard-coded tricks ,among common brute-forcing .

No need to mention that non of these automated tools are ultimate answer for a customer trying to audit and assess his web-app . IMO web-app scanners are not still effective enough to be able to leave them scan target and give back results we can trust on . They`re not even close to this point ! The truth is that they will never reach this golden point . Web 2.0 technologies as an example , made many of these scanners useless and ineffective , and I`m not going to include tricks and mechanisms recently being used to defeat automated scanner and attack tools . AppScan had some honors on being the only automated scanner for scanning AJAX technology but farther checks showed that it was just a marketing buzz , and it`s scanning methodology is still very poor .

As you saw in SPI`s evaluation paper , from beginning lunching the scanner till analyzing final reports , there must be some knowledgeable expert guiding the scanner , optimizing it in every scan step and finally confirming findings to check for false positives . Even at final step you can`t be sure everything is fully tested and you must repeat whole of audit process manually , to reveal vulnerabilities that can not be discovered by automated tools . Of course this last part is for those who have enough budget to hire an expert and look for something more than just an automatically generated report with tens of junk notes being injected into final report.

Here`s how I usually relay on my automated tools in an assessment targeting web-applications:

Step 1: Try to fingerprint web-server and deployed web-applications by lightly queering application and server and also using common online services such as NetCraft.
Step 2: Try to learn as much as possible about target domain/site/web-app by looking at indexed contents by Google and other search engines. Google is really enough , but I never relay only on it.
Step 3: Try to learn the logic of web-app by grouping step-2 findings and manual browsing web-app , and slightly manipulating parameters.

Step 4: Here`s where I switch to selected automated scanner (based on findings) , if it should be used at all ! I`ll optimize settings and include customizations ,and some times develop custom scan/check modules for scanner, if I notice that current scanner checks are not covering what I`ve found in previous steps . Finally I`ll define new scan policy to include/exclude checks available in vulnerability database .
Three points here .
Point One: There`s really no reason to use automated scanners at every assessment you do unless it`s really required .
Point Two: You must know the scanner you`re using very well , and be able to use all of it`s functionality . Not just following default settings and using provided scan policies.
Point Three: Do not relay on only one automated scanner (commercially or free) ,and do not try to waste your time customizing the scanner. Some times there are simply better choices out there for the purpose you`re trying to customize your scanner for. It`s highly dependent if point two apples to you .A good example is auditing web-sites with very limited dynamic contents or completely static contents . Light free scanners for crawling and brute-forcing directories are much more effective than your giant commercial scanner . This is a sad truth while you remember amount of $ spent for product :) SensePost tools are my favorite replacements at this step.
Step 5: Checking and confirming automated-scanner findings (if used) and identifying missed checks and poorly audited functions .
Step 6: Beginning manual auditing (raw traffic/parameter inspection and manipulation) in order to check for vulnerabilities that are usually missed by automated scanners , or not possible to detect them by scanners at all . Session management and handling vulnerabilities , or flaws in logic of application are good examples for this point. Downloading and auditing source-code of identified web-applications (if available) also fits in this step.
Step 7: Documenting findings and exploits , and preparing initial report. repeating step 2,3 and 6 to discover more items, considering gained knowledge about target.
Step 8: Documenting and preparing final report.

Of course these steps are for a black-box assessment . In other scenarios things really change.

I`m going to finish this post with a well-known sentence :
"Security tools and applications are as effective and good as the person using them."