A Portable Web Server on a USB Stick

While doing some recent work on developing a WordPress theme for a friend, I found myself in need of a portable web server; something I could plug into a computer, start up, and continue my development work. Since this was for WordPress I needed PHP and a database. With no sense of whether or not this would be possible. I charged in.

I began by collecting the components.

• Apache
• PHP
• WordPress
• PostgreSQL
• PostgreSQL for WordPress
• Visual C++ Redistributable for Visual Studio 2012

I chose PostgreSQL simply because it was a 50mb download while MySQL was some 300mb. Also, I’d never used PostgreSQL so, why not? Then I found out WordPress doesn’t support PostgreSQL out of the box, but quickly found PostgreSQL for WordPress.

I avoided installers and took the appropriate ZIP file for my distribution so I could simply unzip it to my thumb drive. I unzipped each in turn into it’s own directory off the root of the thumb drive giving me the following directories:

• Apache24
• pgsql
• php
• wordpress

Apache

First up, getting Apache up and running. This is pretty straightforward except that I would need to use relative paths instead of absolute paths if this was going to be a truly portable solution. I didn’t find much online in the way of what relative paths in httpd.conf would be relative to, so trial-and-error it was. The answer: it depends. Some of the paths are relative to the directory where httpd.exe lives while others are relative to whatever ServerRoot is set to in httpd.conf. ServerRoot itself is relative to httpd.exe’s location and this is typically set to one directory up from the Apache bin directory.

ServerRoot "../"

Next is setting DocumentRoot. I stayed with the default htdocs directory under ServerRoot and the DocumentRoot directive is relative to ServerRoot. However the Directory directive is relative to httpd.exe. Don’t forget this one.

DocumentRoot "htdocs"
<Directory "../htdocs">

And with those changes in place, Apache started right up!

PHP

PHP probably presented the biggest challenge as I started to get buried under DLL dependencies because PHP’s directory was not in the PATH environment variable. I could append that path on-the-fly using a batch script, but I wanted to avoid that if possible and, with some jumping through hoops, it is possible. First is what I added to the end of my httpd.conf to get PHP loaded.

LoadFile "../php/libpq.dll"
LoadModule php5_module "../php/php5apache2_4.dll"
<IfModule php5_module>
    PHPIniDir "../php"
    AddHandler php5-script .php
    DirectoryIndex index.php index.htm index.html
    Alias /wp "../wordpress"
    <Directory "../../wordpress">
        AllowOverride All
        Require all granted
    </Directory>
</IfModule>

Using Apache’s LoadFile directive helps solve any missing DLL errors from PHP when starting up Apache. I’ve included the one DLL I got a missing DLL error for. Should you encounter others make similar use of the LoadFile directory to fix things up. All the relative paths here are relative to DocumentRoot except for the Directory directive as seen previously.

On the PHP side of things I made a copy of the supplied php.ini-development file and renamed it php.ini. I uncommented extension_dir and the two extensions for PostgreSQL.

extension_dir = "ext"
.
.
.
extension=php_pdo_pgsql.dll
extension=php_pgsql.dll

But then I ran into some problems. PHP wasn’t finding those pgsql DLL files. I used Process Monitor to see exactly what paths were being looked at and found there were all relative to httpd.exe. I modified extension_dir to be relative to httpd.exe, but I was still receiving the same DLL errors. I second check of ProcessMonitor confirmed not only was the right path being accessed, but the file was actually being opened. What gives?!

Out of frustration, I eventually created an “ext” directory in the same directory as httpd.exe and copied the DLLs over into that directory and, for reasons I’m not entirely sure of, Apache started up without a problem and PHP was working. If you have any idea why this is and what I can do to make it so I don’t have to create this ext directory, please let me know. For now all I care about is that it worked.

DLL Hell

The first time I booted up Apache with PHP on my portable server I got an error about a missing DLL, msvcr110.dll. This is part of the Visual C++ Redistributable for Visual Studio 2012 which can only be downloaded as an installable, not a ZIP. To get this missing DLL I installed the runtime and then copied the DLL from my C:\Windows\System32 directory over to my Apache24\bin directory.

That fixed the issue, eventually…

Because if you’re mixing 32 and 64 bits then the location of this DLL after install might be different. If you’re running the 64-bit version of Apache and PHP, then you need the 64-bit DLL which, on a 64-bit Windows system, is located in C:\Windows\System32. If you’re running the 32-bit version of Apache and PHP on a 64-bit system, the 32-bit version of this DLL is in C:\Windows\SysWOW64. On 32-bit system it’s in C:\Windows\System32.

Good luck with that one.

PostgreSQL

I hadn’t worked with PostgreSQL before and had no clue how to create a database let alone start and stop it. I’m still not sure I do, but here’s what I did. First I needed to create a database. I opened up a command prompt and positioned myself inside pgsql directory and ran this command:

bin\initdb.exe --username <username> --pwprompt -A md5 -D data

This creates a data directory where all information and data about your database server will be stored. You could also think of it as initializing a database server instance. <username> will the the username of the superuser for the database. This can be anything you want. I used “postgres”. During the creation process you will be prompted to provide a password for the superuser account. Note: if you do not provide the username PostgreSQL will use the username of the account you’re logged into Windows under.

When the initialization completes you’ll be told exactly how to start the server.

"bin\pg_ctl" -D "data" -l pgsql.log start

This will start PostgreSQL as a background process. To stop it, use the same command, but change “start” to “stop”. With the database up and running, create a user for WordPress. This can also be done from the command line as follows:

bin\createuser.exe -U <superuser> -W -P wordpress

This creates a user called “wordpress”. You will be prompted to enter the password for the account. Don’t forget it.

Next you need to create the database that will store the data for WordPress. Again, from the command line:

bin\createdb.exe -O wordpress -U <superuser> -W wordpress

This creates a database called “wordpress” and assigns the user “wordpress” as the database’s owner.

We’re nearly there. Just one more step to go.

WordPress

With Apache and PostgreSQL running (start up Apache just by running httpd.exe from the Apache bin directory) it’s time to get WordPress running. In the configuration for Apache I gave above I alised my WordPress directory to “/wp” so open up a browser and go to http://localhost/wp. If you haven’t already installed PosgreSQL for WordPress you should see an error message that PHP is missing the MySQL extensions. No problem!

PostgreSQL for WordPress

You’ll find a readme.txt file inside the PG4WP ZIP file which will walk you through the process. Basically take the pg4wp directory that’s in the ZIP file and copy it into the wp-content directory of your WordPress install. Then make a copy of the db.php file located in the pg4wp directory and place it one level up, inside the wp-content folder. Your folder structure will look something like this:

• wordpress
  • wp-admin
  • wp-content
    • pg4wp
    • plugins
    • themes
    • db.php
    • index.php
  • wp-includes

With that taken care of, go back to your web browser and go to http://localhost/wp. You’ll now be warned about a missing wp-config.php file. You can try the web interface or just make a copy of the wp-config-sample.php file and name it wp-config.php. You’ll need to then edit your newly created wp-config.php file to define the database name (wordpress), the database user (wordpress), and the password. Save the file and reload http://localhost/wp.

If all goes well you’ll finally see the WordPress install page. You will also (most likely) see a PHP warning message at the top about pg_query having failed. Ignore it. Fill out the form and press that install button. With luck you’ll get a success message. You’ll also get yet more PHP warnings. Again, ignore them, it’s okay. Login to your new, minty-fresh, WordPress site.

Starting and Stopping With Ease

At this point everything is up and running and everything is portable! All that’s really left is to find some way to easily start and stop the server with the click of a button. To do that I use a batch script.

@echo off
pgsql\bin\pg_ctl.exe start -D pgsql\data -l pgsql\pgsql.log
cd Apache24\bin
httpd.exe
cd ..\..
pgsql\bin\pg_ctl.exe stop -D pgsql\data

This will open a command prompt window and start PostreSQL first, followed by Apache. The window will remain open while the server is running. When I’m finished bring the command window up and press CTRL-C to tell Apache to shut down. I’m then prompted whether or not I want to terminate the batch job. I say NO which lets the script continue and shut down the PostgreSQL database.

A couple things to note. I found that I needed to be within the Apache24\bin directory before running httpd.exe to get around some missing DLL errors. I also have found that PostgreSQL seems to shut down some times for reasons I don’t know. When this happens I just stop and restart the server and everything is back up and running.

Finale

Congratulations, you have a portable web server. Go forth and develop!

In The Zone With Time

The account-expires attribute in Active Directory carries a value of time in 100 nanosecond units since January 1, 1601 00:00 UTC. In fact quite a lot of Microsoft products are now using this epoch. Java and Unix use January 1, 1970 00:00 UTC as their epoch.

This is important information if you’re ever going to develop code in ColdFusion that needs to, for example, figure out whether or not a domain account has expired.

Another piece of information that’s handy to have is an understanding of how ColdFusion handles large numbers. Numeric values are handled as signed 32-bit values that can range from -2,147,483,648 to 2,147,483,647. ColdFusion can handle much larger numbers, but it does so by converting numeric values into strings and then using its own library to handle math operations based on these strings. And it stores these large values in scientific notation. The problem with this is you start to lose precision. For example, if you run the following CF snippet:

<cfoutput>#( 2147483648 * 2147483648 )#</cfoutput><br>
<cfoutput>#( 2147483648 * 2147483648 + 1)#</cfoutput>

The output is the same for both operations, “4.61168601843E+018“. This loss of precision makes it impossible to calculate the exact time account-expires refers to. So what else can we do?

One trick that sort of works is to divide the value in account-expires
by 10000000 to convert from 100 nanoseconds to seconds. This will significantly shorten the number. So let’s try that on an account-expires value of “129247307050000000” (that’s 7/27/10 @2:58.25 EDT). The line of CF looks like this:

<cfoutput>#DateAdd( "s", ( 129247307050000000 / 10000000 ), "01/01/1601 12:00 AM")#</cfoutput>

Try to run this code and you get the error message “Cannot convert the value 1.2924730705E10 to an integer because it cannot fit inside an integer.”. It turns out that DateAdd() can’t handle a number larger than 2147483647 (the limit of a 32-bit signed integer). 2,147,483,647 seconds is about 70 years of time. To use DateAdd() we need to make the account-expires value much smaller. We could, for example, calculate the account-expires value with respect to a different epoch, one that’s within 70 years of the time referenced by account-expires. We could use any date we want, but I’m going to stick with Java’s epoch of January 1, 1970 to keep things in-line with what’s probably the most common epoch among computer systems.

The number of seconds between January 1, 1601 and January 1, 1970 is 11,644,473,600. If we plug this into the code above we get this:

<cfoutput>#DateAdd( "s", ( 129247307050000000 / 10000000 ) - 11644473600, "01/01/1970 12:00 AM")#</cfoutput>

Run this and we get a result! Specifically {ts '2010-07-27 19:58:25'}. Remember this is in UTC, we’ll need to convert to our local timezone. We’ll do that in just a moment. But first…

You should have recognized by now that this solution won’t work forever. It will fail whenever the data in account-expires is after January 19, 2038. Perhaps this doesn’t bother you since whatever applications you’re developing now will surely be either obsolete or running on a more modern ColdFusion system where DateAdd() can deal with larger numbers before account-expires with that large a value are in use.

Perhaps. But I hate making assumptions like that. Plan for the future! Assume nothing!

If we want lasting code free of the 2038 limitation we’ll need to steer clear of ColdFusion date and time functions. We can do this by using Java objects, specifically java.util.Calendar to manage date and time, and java.math.BigInteger to handle the very large numbers we’ll be working with.

The process is fairly straightforward, but there are a few things to watch out for. Java time is measured in milliseconds, not 100 nanoseconds, so we’ll have to do some conversion. Java and Microsoft’s epochs are different. Calendar objects with a date and time earlier than Java’s epoch will have a negative millisecond value. And we need to calculate this date/time with respect to UTC (GMT timezone) not the local timezone.

First create a BigInteger object. Initialize it to the value of account-expires, and then convert it to milliseconds as that’s the unit of time Java likes to work with.

01: <cfset variables.bigInt = CreateObject( "java", "java.math.BigInteger" ) />
02: <cfset variables.expTime = variables.bigInt.init( JavaCast( "String", "#arguments.accountExpires#" )) />
03: <cfset variables.expTime = variables.expTime.divide( variables.bigInt.valueOf( "10000" )) />

You can initialize BigInteger objects with strings and numerical values. I’m specifically casting account-expires as a string so that ColdFusion doesn’t treat it like an integer and try to put it into scientific notation. Note in line 3 the use of BigInteger’s valueOf() function. This takes a string and returns a BigInteger value. Math functions for BigInteger objects require BitInteger arguments, thus we need to pass all values (even static ones) through valueOf() instead of JavaCast().

Next we set up the calendar object. This means creating a java.util.TimeZone object, a java.util.Calendar object, and then initializing the calendar to the GMT timezone. Once the calendar object is initialized we set it to Microsoft’s epoch.

04: <cfset variables.tz = CreateObject( "java", "java.util.TimeZone" ) />
05: <cfset variables.Calendar = CreateObject( "java", "java.util.Calendar" ) />
06: <cfset variables.gCal = variables.Calendar.getInstance( tz.getTimeZone( "GMT" )) />
07: <cfset variables.gCal.set( 1601, 0, 1, 0, 0, 0 ) />

Note that the second argument of the calendar object’s set() function is 0. Java’s calendar object’s month index begins with 0 for January instead of 1. This is something to be aware of whenever dealing with Java calendar objects.

This next step requires a little explanation. variables.expTime is now in milliseconds thanks to line 3. I’m going to add the value of the calendar, in milliseconds, to variables.expTime. Because the calendar is set to a date before Java’s epoch it will actually be a negative number. The addition operation is actually a subtraction operation. The resulting value of variables.expTime will be the number of milliseconds since Java’s epoch.

08: <cfset variables.expTime = variables.expTime.add( variables.bigInt.valueOf( "#variables.gCal.getTimeInMillis()#" )) />

All we need to do now is reset the calendar to the value we now have in variables.expTime, then convert it to our local timezone and we’re done!

09: <cfset variables.gCal.setTimeInMillis( variables.expTime.longValue() ) />
10: <cfset variables.gCal.setTimeZone( tz.getDefault() ) />

Last, but not least, we convert the calendar object into a ColdFusion date/time object with ColdFusion’s CreateDateTime() function.

11: <cfset variables.expDateTime = CreateDateTime(
variables.gCal.get( variables.gCal.YEAR ),
variables.gCal.get( variables.gCal.MONTH ) + 1 ,
variables.gCal.get( variables.gCal.DAY_OF_MONTH ),
variables.gCal.get( variables.gCal.HOUR_OF_DAY ),
variables.gCal.get( variables.gCal.MINUTE ),
variables.gCal.get( variables.gCal.SECOND )
) />

Note that I’m adding 1 to the month variable because in Java 0 = January, but in ColdFusion 1 = January.

And there you have it. Converting from Microsoft’s 01/01/1601 100 nanosecond based timestamps to something a bit more usable in ColdFusion.

Of course you’ll need a reverse of this process as well, and I’ve already got that for you:

01:  <cfset variables.tz = CreateObject( "java", "java.util.TimeZone" ) />
02:  <cfset variables.Calendar = CreateObject( "java", "java.util.Calendar" ) />
03:  <cfset variables.gCal = variables.Calendar.getInstance( tz.getDefault() ) />
04:  <cfset variables.gCal.set(
Year( arguments.Date ),
Month( arguments.Date ) - 1,
Day( arguments.Date ),
Hour( arguments.Date ),
Minute( arguments.Date ),
Second( arguments.Date )
) />
05: <cfset variables.bigInt = CreateObject( "java", "java.math.BigInteger" ) />
06: <cfset variables.expTime = variables.bigInt.init( JavaCast( "String", "#variables.gCal.getTimeInMillis()#" )) />
07: <cfset variables.expTime = variables.bigInt.divide( variables.bigInt.valueOf( "1000" )) />
08: <cfset variables.expTime = variables.expTime.add( variables.bigInt.valueOf( "11644473600" )) />
09: <cfset variables.expTime = variables.expTime.multiply( variables.bigInt.valueOf( "10000000" )) />
10: <cfreturn variables.expTime.longValue() />

You should be able to follow along based on my explanation earlier in the article. However there are a couple extra bits I’d like to point out.

The calendar is initialized to the local timezone. We don’t have to worry about converting back to UTC/GMT because the calendar function getTimeInMillis() will give you the number of milliseconds from Java’s epoch, which is in UTC.

On line 8 I’m adding the number of seconds from Microsoft’s epoch to Java’s epoch. You’ll also notice the line before that I’m dividing by 1000 to convert time from milliseconds to seconds and after I add the epoch difference I’m converting to 100 nanoseconds by multiplying by 10000000. So why not skip the conversion to seconds and simply add three more zeroes to the end of the epoch difference (making the time in milliseconds) and multiplying by 10000?

The reason is that there’s a problem with precision and the resulting value will vary by a few hundred milliseconds. Since the times I’m working with are in seconds, those extra milliseconds could be ignored. But by dividing, adding, and then multiplying like this I don’t get the millisecond variances. I’m sure there’s better stuff on the ‘net to explain precision with BigInteger objects if you’re inclined to investigate further.

Before you go I have one piece of information I learned while working on this particular topic.

ColdFusion treats all date and time values as if they are in the local timezone. Specifically, if your timezone has daylight savings then all dates are treated as if they have daylight savings, including GMT/UTC times which do NOT have daylight savings time. DateConvert() will not protect you from this issue. There’s a deeper explanation here.

My solution is to check if GetTimeZoneInfo().isDSTon is TRUE. If it is then I need to add (or subtract) 1 hour from my UTC time. Note that this is only needed when converting between UTC and local timezones with ColdFusion date and time functions. The above code specifically stays away from ColdFusion date and time functions so this isn’t a problem with my example. But it is something to keep in the back of your head when you start to notice dates are being calculated an hour off the time they should be.

A bit of vinegar to go with the SOAP.

Not long ago I wrote a 3-part series on using SOAP over HTTPS with ColdFusion. My final solution was to create Java objects directly, bypassing ColdFusion’s CFHTTP tag.

I have since found a subtle flaw with this implementation.

It’s not in the code, but in the  JVM. ColdFusion 8 ships with an older JVM. I recently upgraded our JVM to a more current version in an attempt to resolve a timezone bug. In doing so my SOAP application stopped working.

A little research led me to this article about a TLS bug in Java that could lead to a man-in-the-middle exploit. It appears the way I’m performing my SOAP operation triggers a TLS/SSL renegotiation when it receives a response from the external server.

The short answer is to add the following line to ColdFusion’s JVM arguments:

-Dsun.security.ssl.allowUnsafeRenegotiation=true

This does resolve the problem, but it apparently leaves the JVM vulnerable to MITM attacks. There is another bit of code in that article which shows how to change the allowUnsafeRenegotiation flag on-the-fly. I added this to my ColdFusion code, but changing the flag didn’t appear to have any effect.

If anyone else has played around with this particular problem I’d love to hear about it.

For now I”ve left the JVM in its vulnerable state as we only make HTTP requests from the JVM for a couple of applications and neither of them carry personally identifiable information.

ColdFusion Regex Backreferences

I want to take a string that contains HTML and increment all heading tags by 1. For example, I want to turn <h3>…</h3> into <h4>…</h4>.

I might be able to accomplish this with a long ReplaceList command. However I decide against this as I’ve run into problems with ReplaceList not replacing items in the order it claims to in the documentation. Plus I would have to hard-code every heading tag (both opening and closing) and that would just look ugly. So let’s try and be clever.

This sounds like a job for regular expressions (regex). I opt for the REReplace function over REReplaceNoCase because REReplaceNoCase seems lazy to me, and it forces me to specifically take case into consideration. I like that. Deal with potential problems with input now instead of waiting for it to become a problem later.

So my search string will be something like “<(/?[hH])([1-6])“. This will search for both opening and closing heading tags and I can use the number of the heading as a backreference (that is, I can refer to it in my replacement string). I’m only search for H1 through H6 because the HTML spec clearly states there are only 6 headings, H7 and so on aren’t valid HTML tags.

So my REReplace line might look something like REReplace( variables.input_string, “<(/?[hH])([1-6])”, “\1#Val( \2 + 1 )#”, “ALL” ).

But this will not work.

The reason being that ColdFusion will evaluate all CF variables before performing regex operations. So the backreference to the number of the heading (the “\2” in the example above) will be interpreted as the literal string “\2”. This will trigger a parsing error.

So the workaround is a bit ugly, but still more elegant than just one big ReplaceList function, and that is to loop from 1 to 5 and perform the regex operations one heading at a time. Except there’s another gotcha here. If my loop starts at 1 and works up to 5, I will replace all H1 tags to H2. In the second loop, all H2 tags, including the former H1 tags, will become H3 tags. In the end my HTML will be all H6 tags. The fix is simple, just loop backwards, from 5 to 1. Easy! The code would look something like this:

<cfloop from=”5″ to=”1″ index=”variables.h” step=”-1″>
<cfset variables.input_string = REReplace( variables.input_string, “<(\/?[hH])#variables.h#”, “<\1#Val( variables.h + 1 )#”, “ALL” ) />
</cfloop>

Simple. Except this code won’t work. All the heading tags get dropped for some reason.”<h4>” becomes “<>”. What’s going on?

Refer back to how ColdFusion will evaluate all variables before performing the regex operation. In the replace string I use the backreference “\1”. But ColdFusion evaluates the Val() operation before performing the regex operation, so the backreference the regex operation sees (for an H1 replacement) is “\12”. The regex operation looks for a 12th backreference, which doesn’t exist, so the replaced string is empty.

In proper regex engines, this can be easily fixed in a few different ways. I could explicitly specify the base of the number (regex engines typically accept octal, hex, and decimal base numbers) with something like \x01 (hex-base). Or I could wrap the number in curly brackets like \{1}. But ColdFusion doesn’t have a proper regex engine, but a rather bastardized regex engine.

I need to separate my backreference (\1) from my replacement variable (#Val( variables.h + 1 )#). A space won’t work because my H4 tag becomes “H 4”, which isn’t an HTML tag. Can’t use HTML entities as this isn’t text to be output to the user, it’s HTML that the HTML engine needs to interpret. So what can I do?

Well, here’s the trick I’ve decided to go with.

Looking at ColdFusion’s “Using backreferences” page I see references to special characters used to make letters uppercase (\u) or lowercase (\l). I also see I can perform this case change over a string of characters by inserting \U or \L at the start of the case change, and then insert a \E where the case change should end. This is what I need. These are single, alpha-character special regex characters that do not generate output by themselves. I can use a “\U\E” sequence to separate my backreference from my variable! So my final code looks something like this:

<cfloop from=”5″ to=”1″ index=”variables.h” step=”-1″>
<cfset variables.input_string = REReplace( variables.input_string, “<(\/?[hH])#variables.h#”, “<\1\U\E#Val( variables.h + 1 )#”, “ALL” ) />
</cfloop>

A quick test shows me I don’t even need the \U, I can stick with just a \E to get the job done. But I’ll keep the \U for completeness.

The reason I start with 5 and not 6 is that I don’t want to create H7 tags (which is not valid HTML) so H6 tags will remain H6 tags.

And now we come to the end of my post. This is where someone posts a comment and points out that my solution is, in fact, unnecessary because there IS a mechanism in ColdFusion’s regex engine to handle such a situation and I just didn’t RTFM as closely as I should have. Well, I truly welcome such a revelation should it exist. If not, perhaps Adobe could look into beefing up their regex engine a bit.

Question for Windows Programmers

I’ve got an application (ColdFusion 8) that was (I believe) compiled with Microsoft Visual Studio. You can configure this application to load third-party DLLs to extend its functionality. When the application calls a specific function within the DLL it passes along a reference to an object (C++) in memory and the DLL is suppose to then utilize the methods in the passed object to talk back to the main application.

I have written a DLL for this application.

If I compile it in MinGW and load it into the application, the application calls the DLL function just fine. I have confirmed this by having the function create and write to a little log file letting me know every time the function is called. However, any attempt to access the passed object’s methods results in, essentially, a crash.

If I take THE VERY SAME CODE and compile it with Visual Studio Express (which is free), the DLL is able to call methods of the passed object without any problem.

Now I get that mixing compilers can lead to problems and what’s going on here probably has something to do with stack alignment or differences between how objects are passed to functions between the compiles, but surely there’s a way to make the MinGW compiled DLL handle the passed object like Visual Studio. Perhaps some switch I need to apply when I compile the DLL?

Is anyone aware of any way to do this? Or at least a link to a proper primer explaining why it’s not possible. About all I’ve found off Google are forum posts by nubs like me being told by others to not mix compilers. At the very least I want some dirty details.

Washing Client Certs in ColdFusion with SOAP – Part 3

Continuing from part 2, I’ve got my SOAP request working, but the solution is limited to the OS platform and I don’t like the security implications. What I would really like is a solution that works without the need of a custom tag that is OS dependent. If only CFHTTP wasn’t broken.

I decided to look at what I already had available; namely Java. ColdFusion runs on top of Java and it’s possible to create Java objects from within ColdFusion.

I’ll spare you the grueling details, but I did get caught up at one point playing around with the Java keystore and importing my client certificate into that keystore. Using things like keytool and OpenSSL I tore apart and reconstructed my client certificate and my Java keystore. Ultimately I didn’t need to do any of that and should have just left the keystore alone.

The path to enlightenment began when I found an example of Java socket operations using  the javax.net.ssl.SSLContext object. The example code I was looking at used this object to create regular sockets, not SSL-enabled sockets. I looked into this object and found a few more examples using it to create SSL connections. A bit of back and forth between these examples and the online Java API spec resulted in the discovery that I could load a PKCS12 file as a keystore and pass that to the SSLContext object then create sockets from that SSLContext. This looked promising.

After a bit of trial and error I finally had something that worked!

Here’s the relevant portion of code. I’ll go line-by-line afterwards.

<cfscript>
  // 01: create input file stream from certificate
  variables.ksf = CreateObject( "java", "java.io.FileInputStream" ).init( variables.cert_file );

  // 02: create keystore object from certificate
  variables.ks = CreateObject( "java", "java.security.KeyStore" ).getInstance( "PKCS12" );
  variables.ks.load( variables.ksf, JavaCast( "String", variables.cert_password ).toCharArray() );

  // 03: create key manager factory out of keystore
  variables.kmf = CreateObject( "java", "javax.net.ssl.KeyManagerFactory" ).getInstance( "SunX509" );
  variables.kmf.init( variables.ks, JavaCast( "String", variables.cert_password ).toCharArray() );

  // 04: create SSL context object using key manager factory
  variables.sslc = CreateObject( "java", "javax.net.ssl.SSLContext" ).getInstance( "TLS" );
  variables.sslc.init( kmf.getKeyManagers(), JavaCast( "null", "" ), JavaCast( "null", "" ) );

  // 05: get SSL socket from SSL context
  variables.factory = sslc.getSocketFactory();

  // 06: connect to the server via SSL
  variables.sock = variables.factory.createSocket( variables.server_address, variables.server_port);
  variables.sock.startHandshake();

  // 07: create input and output streams to read from and write to the socket
  variables.sout = variables.sock.getOutputStream();
  variables.out = createObject( "java", "java.io.PrintWriter" ).init( variables.sout );
  variables.sinput = variables.sock.getInputStream();
  variables.inputStreamReader = createObject( "java", "java.io.InputStreamReader" ).init( variables.sinput );
  variables.input = createObject( "java", "java.io.BufferedReader" ).init( variables.InputStreamReader );

  // 08: send the HTTP request over the socket
  variables.out.println( Trim( variables.request_header ) );
  variables.out.println();
  variables.out.println( Trim( variables.soap_envelope ) );
  variables.out.println();
  variables.out.flush();

  // 09: read the response from the server
  variables.result = "";
  do {
    variables.line = input.readLine();
    variables.lineCheck = IsDefined( "variables.line" );
    if ( variables.lineCheck ) {
      variables.result = variables.result & variables.line & Chr(13) & Chr(10);
    }
  } while ( variables.lineCheck );

  // 10: close the connection
  variables.sock.close();
</cfscript>

So let’s work through this.

01: The variable variables.cert_file contains the full path to the client certificate. This line opens that file for reading.

02: This creats a java.security.KeyStore object that expects a PKCS12 format and then loads the client certificate into the keystore. Note that the second parameter of the KeyStore.load() method is the password of the PKCS12 file. You must pass this as a character array and not as a string, which is why the password must be cast as a String object whose toCharArray() method is called to obtain that character array.

03: This creates a javax.net.ssl.KeyMangerFactory object which is used by the SSLContext object to deal with the certificates in the keystore. Note that the second parameter of the init() is the password to the certificate in character array format. You must provide the password twice, once when creating the keystore, and then again creating the key manager factory.

04: Here we create the javax.net.ssl.SSLContext object. The getInstance() call tells what protocol to use with this object, in this case “TLS“. (Note: “SSLv3” also works. No idea which one I should be using. If one fails, try the other.) Once the object is instantiated we initialize it with a key manager from the key manager factory. The second and third parameters of the init() method could point to a trust manager and a source of randomness. It doesn’t appear that either is needed (this is what I saw in other examples) so I pass null values. Note that you have to use JavaCast() to pass a null.

05: Get a socket factory object from the SSLContext object. This factory will be used to create individual sockets using the parameters established with the SSLContext object.

06: Create a single socket from the socket factory by passing the server address (variables.server_address) and port (variables.server_port) we want to connect to. Then call startHandshake() which will create the connection to the server using SSL.

07: Create objects needed to read from and write to the socket we’ve created.

08: Here we send our SOAP envelope to the server. Note that since we’re doing everything by hand we also have to send the proper HTTP headers. Both the variables.request_header and variables. soap_envelope variables were created using the CFSAVEDCONTENT tag as shown in part 1. You want to create the envelope first so you know the correct content-length value for the HTTP header. Since I decided to stick with SOAP 1.1 the header also contains the SOAPAction field and the content-type is set to text/xml.

Here is an example of how I created the headers:

<cfsavecontent variable="variables.request_header">
POST <cfoutput>#variables.server_request_path#</cfoutput> HTTP/1.0
Host: <cfoutput>#variables.server_address#</cfoutput>
User-Agent: SOAP Washer/1.0
SOAPAction: "<cfoutput>#variables.soap_action#</cfoutput>"
Content-Type: text/xml
Content-Length: <cfoutput>#Len( Trim( variables.soap_envelope ))#</cfoutput>
</cfsavecontent>

Standard HTTP/1.0 headers. You can make the User-Agent field just about whatever you want.

Also worth mentioning is that I didn’t try to combine the two (HTTP headers and envelope) into a single variable. This is because how CFSAVEDCONTENT handles newline characters is different between ColdFusion 8 and ColdFusion 9. By keeping them separate this code is able to operate under either version.

The empty println() commands are there to send a newline. And the flush() call simply sends anything left in the buffers, thus completing the sending of our request.

09: This is where we read the response from the server. The response can only be read one line at a time, so we need a loop to keep reading one line at a time and appending it to the variable where we’ll store the entire response (variables.result).

A tricky thing working between ColdFusion and Java objects is that if you assign the return value of a Java object method to a ColdFusion variable and that return value is null then the ColdFusion variable is unset or deleted. Since the readLine() method returns null when there’s nothing more to read, the only way to know we’ve completed reading the response is to test if variables.line, where we store the return value of readLine(), still exists. If it doesn’t, we’ve reached the end of the response.

10: Close the connection

The ColdFusion variable variables.result will contain the entire response from the server, including the response HTTP headers. Meaning if you want to get a proper XML object from xmlParse() you’re going to need to first strip away those headers.

<cfset variables.temp = REFindNoCase( "content-length: ([0-9]+)", variables.result, 1, "True" )>
 <cfif variables.temp.len[1] GT 0>
  <cfset variables.length = Val( Mid( variables.result, variables.temp.pos[2], variables.temp.len[2] ))>
  <cfset variables.xmlResult = Trim( Right( RTrim( variables.result ), variables.length ))>
</cfif>

This should do the trick. Although you may want to throw in some code to strip out the status code to ensure a good transaction.

And there you have it. Handling transactions with a third-party web site using client certificates under ColdFusion using only native ColdFusion objects.

Washing Client Certs in ColdFusion with SOAP – Part 2

In part 1 I introduced you to basic SOAP consumption in ColdFusion. Let’s see where things go from there.

The task at hand was to integrate our system with a third-party site. I authenticate users locally, request a token from the third-party site that allows the user to access said site, and then redirect the user to that site with the token passed on the URL.

But how does the third-party site know to trust my requests and not the requests from other people? After all you don’t want people forging requests to access the third-party site as some other user. The solution for this third-party site is client certificates. A client certificate is either issued by the third-party site or you provide your public certificate to the third-party site and the third-party’s site is configured to trust the certificate. SOAP requests are then made over SSL using this certificate to confirm your identity and to encrypt the channel.

Starting with ColdFusion 8 CFHTTP accepts two parameters to afford the use of client certs, clientCert and clientCertPassword. The clientCert parameter points to a PKCS12 formatted file containing your public and private keys and, possibly, the certificate chain from the root certificate authority (such as VeriSign or Thawte) down to whoever issued your client certificate. The clientCertPassword parameter contains the password with which the PKCS12 file is encrypted.

A note to ColdFusion developers:
The PKCS12 must be encrypted with a password! For whatever reason (I believe a limitation in the underlying java.security.KeyStore object) your cert must have a password. This is never explicitly stated in the ColdFusion 8 documentation.

I obtained my client certificate and set out to start writing code to talk with this third-party.

The first problem I encountered was I would not be able to use the CreateObject() method covered in part 1. The reason being that there was no way to provide my client certificate to the object. So it’s back to the CFHTTP method.

The second problem I encountered had nothing to do with ColdFusion and everything to do with the documentation provided by the third-party. Turns out the header for the SOAP request changed considerably during development. I had been given some early development documents that did not reflect the current header structure. Once I realized the problem and obtained the current documentation I was able to correctly construct my SOAP request’s envelope header and…

I got another error.

But this time it wasn’t a normal SOAP request error. The CFHTTP object’s filecontent value contained nothing more than “Connection failure”. But the HTTP status code was 200, which indicates a successful request. Previous SOAP request errors would return a 403 status code. This was odd.

In searching the Adobe forums and the internet in general I found sparse comments about possible problems with CFHTTP handling SSLv3 sessions, although there wasn’t any sort of official comment or response to the few reports of this problem. I loaded up the developer edition of ColdFusion 9 on my own computer to see if perhaps this problem had been resolved with the latest copy of ColdFusion. It had not.

To confirm this as a problem with ColdFusion and not my client cert, or my SOAP request, I installed Apache and PHP locally and ran the equivalent PHP code. The PHP code worked perfectly. I started trying to do as much comparison between the two platforms as I could. I event went so far as to run a packet capture on the PHP and CF requests (pointing them at a dummy, local page that wasn’t encrypted so I could see the requests) and compared them to make sure everything was the same, which they were.

Eventually I found a post online that mentioned CFHTTP wasn’t up to the job, but a third-party custom tag written in C++ did work just fine. That custom tag is CFX_HTTP5. I downloaded a demo copy and installed it locally. How CFX_HTTP5 handles client certs is different from CFHTTP. Rather than simply pointing the tag at the client cert, I had to install the client cert into the Windows certificate store and then point the tag at the store. There is a bit of work involved with it, although nothing too difficult and it’s all covered in the CFX_HTTP5 documentation.

Once I had the tag and the certificate imported into the local Windows certificate store, I rewrote the CFHTTP call using CFX_HTTP5 and it worked! The SOAP envelope was the same, the headers were the same, the only difference between the two was the logic underlying the tags.

Something is broken with CFHTTP and it can’t be used to do some operations using client certificates. But at least there is an alternative.

However I didn’t like the alternative.

First, it’s a Windows-only solution. We’re out of luck if we’re running ColdFusion on a Linux machine.

Secondly, the client cert must be stored in a place that the ColdFusion process has permissions to access, and it is accessed without needing to know the client cert password. The result is that any person with permission to create CFM scripts that are executed under this process could authenticate against this third-party web site. In a shared hosting environment this can create a serious security issue. The only solution is to separate the process, but that probably means a separate server and a new OS license and hardware costs. If that option isn’t available you have some problems. You might register the CFX with a name that contains random characters and hope none of the users in the shared environment know how to enumerate registered custom tags. I’m not sure if that is possible, but I’m willing to bet it is.

Long story short, I don’t like the CFX_HTTP5 solution.

So, in Part 3, I go back to the drawing board.

Washing Client Certs in ColdFusion with SOAP – Part 1

Recently I was asked to look into integrating our systems with an external application via the third-party’s single sign-on system. The way it works is simple enough. We would have an application that authenticates the user through our system and then sends a request to the third-party asking for a token to sign the user into their system. The third-party would return the token that I then give to the end-user and redirect them to the third-party web site. This token is how the third-party would authenticate the user into their system. (The token is nothing more than a long string of characters that is passed on the URL of the redirect.)

The method of obtaining the token is also fairly simple. The application submits a SOAP request over an SSL session to the third-party’s authentication server and that server would respond with the token. SOAP, for all the technical specs and other crap, is very straightforward. It’s a simple XML document consisting of a root element called ENVELOPE which contains two children called HEADER and BODY. The header isn’t always required and the body typically contains elements with the names of various fields the SOAP function your calling requires with each element containing the value of that parameter. Very straightforward.

A SOAP request in ColdFusion couldn’t be simpler, especially with the CFSAVECONTENT tag. Simply construct your envelope inside a CFSAVECONTENT tag and then use CFHTTP to submit the request. It looks a little something like this:

<cfsavecontent variable="variables.soap">
<?xml version="1.0" encoding="utf-8"?>
<soap:Envelope xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/">
  <soap:Body>
    <GetCurrentTime xmlns="http://ws.historicaloptiondata.com/" />
  </soap:Body>
</soap:Envelope>
</cfsavecontent>

<cfhttp
  url="http://ws.historicaloptiondata.com/Service.asmx"
  method="POST"
>
  <cfhttpparam type="header" name="SOAPAction" value="""http://ws.historicaloptiondata.com/GetCurrentTime""" />
  <cfhttpparam type="header" name="Content-Length" value="#Len( Trim( variables.soap ))#" />
  <cfhttpparam type="xml" value="#Trim( variables.soap )#" />
</cfhttp>

<cfdump var="#xmlParse( cfhttp.filecontent )#" />

You should be able to plug this code into a CFM file and run it without having to touch a thing and you should see the CFDUMP of an XML object. (I say “should” because xmlParse() seems to first try and open a file with the name of the content of the passed variable and, when that fails, treat the passed value as an XML document itself. This can trigger errors and make it unusable if you employ any sort of file operation restrictions on your server. In which case modify the code to remove the xmlParse() call and just dump the cfhttp.filecontent.)

A few notes about this code SOAP in general.

• This is an example of a SOAP 1.1 request. There is another, slightly different format known as SOAP 1.2. The major differences between the two are that the content-type for 1.1 is text/xml, but for 1.2 it is application/soap+xml. Also the SOAPAction HTTP header is no longer needed in 1.2.
• The SOAPAction HTTP header’s value must be wrapped in double-quotation marks. And sometimes the first character after the open quotes will need to be a pound (#) symbol. This means ColdFusion programmers will need to be certain they escape these special characters in their values.
• If the CFHTTPPARAM type “xml” is present, CFHTTP automatically sets the content-type to text/xml. I am not sure if it’s possible to override this, but I believe not, therefore you’re almost always going to have to stick with SOAP 1.1 if you’re using CFHTTP for your SOAP requests.
• You must Trim() the variables.soap variable! The newline at the beginning of the value, which exists because there is a newline immediately after after CFSAVEDCONTENT tag (for visual formatting purposes) will make the XML document you’re sending an invalid XML document and result in errors.

Now comes the WSDL file. A WSDL file is an XML documents that describes the functions and parameters of said functions available through a SOAP service. The CreateObject() function has a “webservice” object type which will consume a WSDL file and create an object with all the available functions offered by the SOAP service. All the stuff with XML and ENVELOPES and CFHTTP becomes transparent and, as it turns out, SOAP can be simpler than my previous example. The above code can be reduced to the following using CreateObject():

<cfset ws = CreateObject( "webservice", "http://ws.historicaloptiondata.com/Service.asmx?WSDL" ) />
<cfset ws.getCurrentTime() />
<cfdump var="#GetSOAPResponse( ws )#" />

This makes this life much simpler for ColdFusion programmers. No need to worry about what version of SOAP you’re using or what URLs you need to submit your request to, the formatting of your SOAP envelope, the SOAPAction HTTP header variable, etc. It’s all taken care of for you by ColdFusion.

So this integration I was asked to do should be a piece of cake, right?

Wrong.

Simple SOAP is simple. Complex SOAP… well, you’ll see in Part 2.

WordPress Exploit – PHP Programming

A recent exploit was discovered for all versions of WordPress prior to and including 2.8.3. The vulnerability is in how variables are checked while processing a password reset request. The offending code is quite simple and by itself looks very innocent:

if ( empty( $key ) )

If the $key variable is empty, an error is generated. If $key is not empty it’s passed to a SQL query that looks up the user whose key is being reset. Seems innocent enough, right?

The variable $key is a sanitized copy of $_GET[‘key’]. And what happens if the value passed on the url for key is an array?

Well the sanitization step is a simple preg_replace, like so:

$key = preg_replace('/[^a-z0-9]/i', '', $key);

The PHP documentation states that if the subject ($key) is an array, the return value will be an array as well. It also states that if no matches are found, the original subject is returned. Since $key is empty to begin with, thus no matches will be found, a copy of $key is returned. Essentially $key remains unchanged. So what happens when you pass an empty array to PHP’s empty() function? Well the documentation says it should return FALSE. So what’s the problem?

The problem is the array isn’t really empty. It contains a single node which contains an empty string. So the empty() call returns FALSE, thus no error is generated for an empty $key.

Surely the SQL check will return no entries since there is no key to search for, right?

Wrong. Here’s the code:

$user = $wpdb->get_row($wpdb->prepare("SELECT * FROM $wpdb->users WHERE user_activation_key = %s", $key));

The $wpdb->prepare()  is used to insert values into SQL statements. $key is treated like an empty string. Therefore the SQL that is generated looks like:

SELECT * FROM wp_users WHERE user_activation_key = ''

And that query will return every user who has an empty activation key — that’s everyone not currently trying to reset their password. Since the first row returned is the record that gets reset, you will almost always be resetting the admin account. If you keep making the same request over and over you will eventually wind up resetting the password of every user in the system!

Reset passwords are auto-generated and then e-mailed out to the users. This sort of attack could be used as a weak style of denial-of-service (or pain-in-the-ass attack as I prefer to call it) or an attacker who has access to a victim’s e-mail account (or can packet sniff their e-mail downloading)  could take control of the blog.

A fix is already on the way as you can see by viewing wp-login.php in WordPress’ CVS. The fix seems simple enough:

if ( empty( $key ) || !is_string( $key ) )

So $key must be a string and it must be empty. Seems very straightforward. Although I would prefer they went the added step of simply modifying the SQL query to check against only those user_activation_code fields that are not empty strings. In other words add something like

$user = $wpdb->get_row($wpdb->prepare("SELECT * FROM $wpdb->users WHERE user_activation_key = %s AND user_activaction_key <> ''", $key));

That would more correctly fix the issue, as it would protect against even those attacks that are somehow able to bypass the initial empty() check.

WordPress appears to also be adding a username to the password reset request so that simply passing an empty key on the URL and hitting reload a bunch of times won’t result in everyone having their password reset — probably something they should have done in the first place.

So there it is. Patch your WordPresses, either by manually editing wp-login.php or waiting for the official patch to come out sometime later this week.

cf_buffer

My knowledge of ColdFusion 8 (or hell, even 7 for that matter) isn’t terribly broad. So if what I’m about to cover can be accomplished with an existing CF function or tag please let me know.

Output Buffering.

It’s something I first got use to with PHP. What it does is allow you to set a start and end point in your program and any output generated between those two points gets put into a variable which you can then manipulate further before outputting the content.

Recently I was asked if this was possible to do with ColdFusion. I couldn’t think of any existing feature that does this so I put together a (surprisingly) short custom tag that would do just that.

Here it is:

<cfif CompareNoCase( thistag.ExecutionMode, "start" ) EQ 0>
  <cfparam name="attributes.buffer" default="variables.buffer">
<cfelse>
  <cfset "caller.#attributes.buffer#" = thisTag.GeneratedContent>
  <cfset thisTag.GeneratedContent = "">
</cfif>


I put this into a file named buffer.cfm. Now anything that occurs between an opening and closing CF_BUFFER tag will be stored in a variable which can then be further manipulated however you want.

I did find something similar using getPageContext().getOut().getString(), but nothing simpler.

You might argue that this feature shouldn’t be needed in the first place. That whatever the need is it can be solved by redesigning the application. That might be true. But it’s a handy feature to play around with.