Microsoft Quality Engineering

Go Microsoft account recovery guys.  I tried every one of their recovery methods including email, call phone, and text phone, none of which worked at all (my phone number is correct as is my email). After waiting for 30 minutes for the code to come I finally got a code from them in an email.  See below what happened.  ;-) Not only can they not actually come through on any of their account recovery methods, but they can't even use the code once they get around to sending it to me.  Quality engineering there.

Solving Problems with Mac OS Server Spotlight Sharing

We have a Mac server for our advertising department that contains tons of images.  They use spotlight on their Macs to search the shared volumes from the server.  There were a few folders that had somehow become stuck and new files and folders in that stuck folder would not be found when searching.

I connected to the server and found that searching on the actual server worked just fine on the problem folders.  This ruled out any spotlight exclusions for those folders since they should never be found in that case.

I decided to check the index status using mdutil

mdutil -as

Surprisingly it showed the stuck folder as having its own index and that it was working fine.  I tried to figure out how to modify the list of folders that have their own indexes and couldn't find anything about it.  The stuck folder did not have a mount independent of the main volume one, so it should not have been there.

Eventually after much searching I found that at the root of each volume inside the .Spotlight-V100 folder there is a config file that contains the configuration for the volume's indexing called VolumeConfig.plist.  For some reason this config file contained special configuration for my stuck
folder.  So I edited it and took out the offending configuration leaving only the block for the root of the volume using:

sudo vi /Volumes//.Spotlight-V100/Store-V1/VolumeConfig.plist

Once that was gone I restarted the indexing service using

sudo launchctl stop com.apple.metadata.mds

Now new files and folders added inside my stuck folder instantly appeared in the search box on a remote computer.  Existing stuff still didn't show up though so I still needed

sudo mdutil -E /Volumes/

to force it to reindex the volume

Now everything is working great!

My assumption is that at some point this folder was shared as its own volume.  This got the folder in the configuration list so that it had it's own index.  Then the volume was deleted but the index remained for some reason.  It seems that spotlight on the local system is aware of the potential for multiple indexes and searches them all, but remote searching only searches the index at the root of the volume.  Also the index util for some reason doesn't index folders with independent indexes into the root index.

I'm sure there are some bugs here, but this seems to resolve the issue.

Enjoy!!

Making Sliced Emails Work Reliably In All Browsers

After spending the last 2 and 1/2 hours fighting my a new email with a complicated table (created by Photoshop) I have determined that the following items are crucial to avoid gaps and alignment problems in Internet Explorer, Chrome, and Safari. 

1. Photoshop's blank 1px spacer table cells at the end of each column and row are ingenious and important.   Don't delete them (like I did) because they keep IE from screwing up the table in email clients.
2. Photoshop puts an accurate height on the containing table which is important for IE which can't add?
3. Wherever you see an alt tag (the images) search and replace in 'border="0" style="display:block"' which makes Chrome and Safari deal with spacing better.

With these three items I think your table will not get screwed up... knock on wood.

Everest's Disco Yeti and Politics of Engineering

I finally watched that great Discovery channel video tonight when I got on a kick to try to see the Yeti in A mode.  After watching the movements, and seeing the mechanism I really don't get it.  This cannot be an engineering issue or a money issue.  This MUST be a political issue.  Doubtless they had tons of money allocated for maintenance on this monster, and to record new movement sequence for this guy that just moved his fingers, eyes and head as shown in this video 

would not put any stress on his "cracked foundation", and would not cost anything near the amount of money they saved by having disco yeti for 5 years.  

This my friends is almost assuridly imagineering pouting that the guys in Florida won't spend the money to fix this right.  So they refuse to put in the half day of reprograming it would take to make this guy move a bit. They can't stand to have their great achievement neutered into a normal functional animatronic.  It's all or nothing! Totally smells of grown adults acting like 5 year olds.  

Now given, their fear may be justified, because if they created limited A mode then operations may never bother to return this guy to his former glory... Still can't they just work out a deal that they will do it for the sake of the show in exchange for a promise that they will fix it once Avatar land comes on line.  It is all about the show right, and not about Joe Rohde's (or someone else's) pride? 

Maybe this is why Disney doesn't seem to let imagineering do animatronics much anymore.   They give them all to outside contractors like Garner Holt from what I have heard of late.   All of Radiator Springs Racers was done by them right? Was little Mermaid internally done?  Maybe her floating hair was?  Sure their group screwed up on Murphy the new Fantasmic dragon temporarily, but at least Disney could blame someone, and not have to baby them to get a fix.

What do you think?  Am I on the right track?

Prevent All Non Desirable Apache Methods

So our security audit claims that I have to shutdown all Apache httpd methods but POST, GET and HEAD.  I went to the Apache documentation and they claim you should use LimitExcept.  Sounds great right, so I tried using it in all the places they allow, but it doesn't work anywhere I put it.   After scouring the web I have just given up and used something really simple:


RewriteEngine On
RewriteCond %{REQUEST_METHOD} ^(?!POST|GET|HEAD)
RewriteRule .* - [F]

Works great (with the minor performance cost)... I wish Apache would just fix LimitExcept so it could be global.  Some of us don't use Directory entries like they expect.

Just thought I'd post though so that others may not have to suffer the same waste of time as myself.

Simple way to do a facebook like check in java

People are making this so hard.  Now facebook is moving from fbml you gotta do the like check yourself instead of using the very convenient . Here is my simple solution after 3 hours of trying the hard versions.

static final Pattern FB_SIGNED_REQUEST_PATTERN = Pattern.compile("liked\":(.)");
static final BASE64Decoder BASE64_DECODER = new BASE64Decoder();
public static boolean isFacebookFan(HttpServletRequest request)
throws Exception
{
String fbreq = request.getParameter("signed_request");
if (fbreq == null) throw new Exception("No request");
fbreq = new String(BASE64_DECODER.decodeBuffer(fbreq));
log.error(fbreq);
Matcher m = FB_SIGNED_REQUEST_PATTERN.matcher(fbreq);
return m.find() && m.group(1).equals("t");
}

The Firesheep Problem, and How rcwilley.com Is Protected

Recently with the new "Firesheep" firefox addon that steals facebook and twitter sessions over unsecured wifi sidejacking is in the news.  I thought I'd sit down and write about my solution to the problem which protects rcwilley.com sessions from being hacked.

A few years ago when I changed jobs into the web engineering business I was forced to get up to speed on session cookies and how they are used.   For those of you not familiar with how cookies work here is a quicks simplified primer:

Cookies are little pieces of data that a website can send to your browser.   Then whenever your browser communicates with the same server it got the cookie from, it sends it in the request.   The web server can then look at this cookie and know what browser he is talking with.  When you log into a website, it is very common for them to send a piece of data called a session cookie to your browser.  Then every time you ask for a new page they know who you are.   Now this is all perfectly secure and safe as long as you never communicate with the server over a non-secure http connection.  However most websites, after using a secure https connection to send your login and password over, switch back to http for performance.  They don't pass your password any more, but every request from your browser has the session cookie sent UNPROTECTED to the server.  That is how the server knows who you are for the rest of your visit.

Now there is a concept of a secure cookie which the browser is only allowed to send over a secure https connection, but they are hardly ever used.  This is because the user may make a request over http all of a sudden, and the server won't know who it is.

One of my first tasks was to create a single sign on kind of solution that allowed customers to log into our site and stay logged in while they moved between normal and secure pages on our site.   I got it working the commonly accepted way of using the session cookie to maintain the users login credentials throughout their visit.  This worked just fine, but I wanted to make sure my solution was secure when we switched between http and https.

After hunting around on the internet on ways to hack sessions I learned about sidejacking.  This is when someone able to watch network traffic between a browser and the web server just grabs a session cookie and uses it to pretend to be someone else.  I asked one of the top engineers at my web consultant firm how to prevent this, and he said I shouldn't worry about it because it was to hard to do a sidejack of a cookie, and no one else worried about it.  I visited many big websites, and watched their cookie usage. None of them used secure cookies.  Facebook and twitter were common examples of these kind of sites, so it seemed that my co-worker was right about sidejacking being a non-issue.

Basically web sites seem to be in the following camps:

1. Sites that are stateless and need no cookies
2. Sites that use https for login, but no secure cookie, risking an http connection sending the cookie in the open.  These are the risky ones like facebook and twitter.
3. Sites like 2  that fortunately make you login again every time you go into a secure area.  This helps.
4. Sites like 2 that make you login every time you enter a secure area, and then use a secure cookie.  Amazon appears to be in this camp.
5. Sites that are completely https and use secure cookies and suffer the performance penalties.  Mostly banks and the like.

My favorite kind of site is 2 because it doesn't irritate the user after they already logged in, but it is too risky.  I still just didn't feel right about being a type 2 site.  So for rcwilley.com I adopted a combination solution that would give me the benefits without the problems.   The simple solution is to use both secure and normal cookies, and require the secure cookie whenever the user re-enters a secure area.  It was very easy to implement and completely solves the problem as far as I can figure.   Twitter and Facebook ought to consider this method if they don't want to go completely secure like a bank.

Now some nitty gritty details for those who care.  When the user logs in, I do it over a secure request and I give them their normal session key in a normal cookie.  I don't have to mess with the default behavior of tomcat at all.  At the same time I also generate a secure cookie and send that as well.  The secure cookie value is simply stored in their session with everything else.  Thereafter they can go to an http section of the site, and back without having to re-login.   Whenever they try to make a secure request from then on I just check to make sure they also sent me the correct secure cookie as well.  Otherwise they have to log in again.  Works like a charm.

IPhone 4's Great New Camera... What is a backlit sensor?

The new iPhone 4 has a great new backlit camera sensor. Most people don't understand why this is nice so I wrote up this little description for the TII Podcast I listen to. A major factor in my recent camcorder purchase was the backlit camera sensor of the Sony camcorders. This sensor severely reduces graininess in lower light situations making it a must have for me.

Here's how it works. Light sensor pixels are not very reliable at low levels of brightness so the camera turns on the backlight to make sure this doesn't happen. Of course this makes the whole scene uniformly brighter, but the camera uses an algorithm to darken it back down again when the light is on so that you don't notice the difference.

This is a very smart implementation of the technique used in the movies for years to get around the same problems with film cameras. When the crew of a movie films a night scene they don't really film it in the dark. Instead they film it on a well lighted set and darken it in post production avoiding the graininess that comes from low light filming.

Just thought this was interesting.

Simplest JQuery ToolTips

Well I think I may have developed the simplest tooltips possible using JQuery. Everything out there seemed too fancy and took too much space, so I just whipped up something super simple. All you need is the amazing JQuery library, this javascript method:

$(document).ready(function(){enableTooltips(verticalOffset, horizontalOffset)}) // Set these values how you like, I used 35, 0

function enableTooltips(topOffset, leftOffset)
{
$('[tooltipText]').bind(
{
mouseover: function() {
if ($('#tooltip').length == 0)
$('body').append('

')
var o = $(this).offset()
o.top += topOffset;o.left += leftOffset
$('#tooltip').css(o).html($(this).attr('tooltipText'))
.stop(true).animate({n:0},1000).fadeTo(300,1)
.animate({n:0},5000).fadeTo(500,0)
},
mouseout: function() {
$('#tooltip').stop(true).animate({nothing:0},500).fadeTo(500,0)
}
});
}

And some css like this:

#tooltip {
font:normal 12px Verdanna, Arial, Helvetica, sans-serif;
position: absolute;
z-index: 1000;
border: 1px solid #111;
background-color: #eee;
padding: 2px;
opacity: 0;
filter:alpha(opacity=0);
}

On the items you want tooltips just put a tooltipText property like this:

<input type="submit" src="ButtonHome.gif" tooltiptext="Go to home">

and you win! JQuery is amazing!

The Apple Flash battle is similar to the USB and Floppy transition

Just a short note today as an iPhone user.

It is painful that Apple has booted Flash out of their phone, and as a user it causes me to suffer a bit, but I think change has to hurt. This is just Apple's thing. They like to force innovation to happen, which has made a crucial difference in consumer electronics and software over the years.

Intel invented USB, but could not get it implemented in Windows computers. I was very irritated when Apple tossed their PS2 like serial ports on all their machines but... I think it did have the needed effect of accelerating USB adoption.

They did the same thing with the floppy transition, when they dropped the drives from all their stuff. It was a bit painful for a while but thumb drives and writable CDs came along quickly and solved that.

Apple is irritating people with this, BUT it is absolutely having the effect of making HTML 5 a reality just as it did with USB, and the floppy. Even Microsoft has put very good support in IE 9. I don't think this would have happened nearly as quickly without the iPhone thing, and as a web engineer I appreciate the push.

I must keep my iPhone jailbroken thanks to stupid Apple limitations, but on most things I thank Apple for making me suffer for a bit. Without them we might still be using DOS.

JSTL is Old Fashioned But Cool

I am not afraid to say that I really like JSTL. It may be old fashioned, but it gets the job done very nicely. Is it just me, or do many of these new frameworks just seem like they don't buy you all that much. Anyway I created this post to shove my favorite JSTL expressions/hacks, toolkits etc. I'll just keep editing it to add more as I think of them/find new ones.

---

Quickly get the current year from JSTL (and jsp)

<jsp:usebean id="now" class="java.util.Date">

<c:set var="year" scope="page" value="${now.year+1900}">

---

I really like the displaytag library. So simple and powerful (especially for internal crud pages).

http://displaytag.sf.net

---

Why does CSS suck so bad? They forget basic things like allowing URLs to break in a table so the table formatting doesn't get thrown out. Here is a snipped to stick hidden breaks in a url so that it will wrap (oh, and still escape xml).... Argh!

${fn:replace(fn:escapeXml(variablename),"/","/")}

New Moon Gets It Right And Rakes In The Dough

Hi there whoever may be reading this. I've been really happy to hear of New Moon's success. Stephenie Meyer really seems like a great person, and I love to see her succeed. The thing that has really surprised me is how much she and the no-name production company Summit Entertainment is succeeding this time. In case you didn't know the movie made $275 million dollars in the first weekend. Third to Dark Knight and Spider Man 3 in it's opening weekend in the US. Top opening ever in China. Amazing.

I mean, put this in perspective. Today Disney announced that Johnny Depp has agreed to take 35 million to reprise his role as Captain Jack. Dark Knight and Spider Man 3 cost around 250 million each to make. New Moon is competing against these movies and cost only 50 million to make. They are making a killing.

Why does this movie succeed where so many others fail, and at such a tiny cost? Well, I can think of 3 reasons that probably have something to do with it:

1. They are relatively clean. Devoid of crass language, potty humor, and overt sex.
2. They target women. 80% of the tickets were sold to women this weekend. Let's face it, women are rarely targeted effectively with the blockbuster movies. Hollywood usually puts much more effort into men.
3. They target the fans! Less people are excited about the Harry Potter movies because the people who love the books (a HUGE number of their viewers) hate the destruction of the stories that happens in the movies. The twilight movies (and the books for that matter) are designed to please the fans! What a revolutionary concept! This makes me wonder what would've happend if the Harry Potter movies had gone after the book fan base. Would they be up there in the top 3 too?

Anyway, I was happy to be one of the men in the 20% this weekend. Sure the movie was a bit cheesy, sure it would be nice if Edward smiled once in a while, sure Jacob could show off his chest a bit less... ;-) But I really enjoyed watching it with my wife. I am happy to see a movie that compliments the book, one that is produced outside the hollywood mainstream, and one that is changing the rules of what movies can succeed in a big way.

I am sure that this will not go unnoticed by Hollywood, and that we are going to see a ton of copycat movies coming out soon that I will have to sit through... Chick flicks are back in vogue, and I may even like it.... Maybe....

Halloween 2009

Well we survived another crazy Halloween season. This year in haunt news:

1. Bought a brand new projector with money from our DVD and music sales. A Panasonic PT-AE3000U which is the one I have been planning to purchase for a few years (at least the older model). It is a very nice full HD projector, and looked great on my house. We watched Johnny Depp in "Secret Window" last night, and it looked great. OK movie by the way. Johnny Depp can make anything fresh.
2. Remixed the whole haunt to 1080p HD. That took most of my time for haunt enhancement this year, but it was worth it. When watching the haunt everyone kept saying "____ is new this year right?" I just kept saying, "no, it was there before, you just couldn't see it ;-)". Hooray for HD! Only problem is that with the 16x9 aspect I had to put the projector further away from my house. I also had to lower it to let me get the sidewalk and the top of the house. This is very unfortunate because people's shadows were much more obvious. Don't know the solution to that one though because any farther back and I will have to cut down my neighbors trees just like I did mine ;-).
3. Built a huge 16 foot projection tower to put my new projector in. I opted to make it out of 3" ABS so the projection box can go up and down like on a flag pole. It worked all right after some battling. Of course it is also nice that ABS is black. Because of the new tower I was able to put the projector whatever height I wanted in the middle of my neighbors yard without having to use the fence for support. Very cool... Very large galvanized pipe stakes holding that baby there... Hope I can get them out.
4. Worked on some new segments for the haunt, but didn't finish in time. Tried them out at the end of the night on family and friends and they were a hit.... Next year!!
5. Got a nice write up in the Taylorsville/Kearns newspaper that got a lot of nice comments. You can read it here: Bates Haunt in Local Newspaper

The video is still in progress. I taped it with my kind brother in law's professional HD camera, so it should look good... Unfortunately I used the P2 compression so he has to convert it to something useable for me.

I thought you might be interested in some stats on my Adobe Premiere project. It may be the most complex Premiere project ever (probably cause no one else is stupid enough to do a show this complex in Premiere ;-) ). I really need to port it to After Effects.

• 43 Video Sequences
• 287 Video Tracks
• 2422 Layered Video Filters
• 4729 Positioning Keyframes

Not sure how long it takes to render because it crashes Premiere unless prerender most of the sequences. If I start from scratch and combine the time for all the individual builds it adds up to 6+6+4+2+5+3+5 = 31 hours. Fortunately the final mix is only 5 hours, so if I am making only minor tweaks at the top level, I can render it overnight.

Hope you all had a Happy Halloween!!!

Wow, it has been a while. Deck of death!

So earlier this summer I decided to take on a large deck with wrap around stairs. We opted to go with Trex Brasilia with their new hidden fastener system because it is a nice dark red without the cheesy grain of normal Trex. The vote is still out as to wether this was a good choice or not. Trex is cool, but it is scarier to work with then wood because there is no sand-out-the-bad-spots, and I am not sure how it will hold up eventually. I have had to be very careful not to scuff or scratch the boards too much, and that kind of scares me. Due to the 30' of wrap around step, I have still not finished the project, but should this week.

Some things I have learned are below:

• After framing your deck, immediately start on the decking. Otherwise pressure treated wood warps like crazy when it dries without the influence of the decking, and then you have to work hard to get stuff to line up again.
• Overbuild your deck structure. Do not come close to maxing the spans of your chosen wood because decks should not have spring at all.
• Be extremely careful when making stringers. I had to make around 28 for my stairs and inaccuracies in my stringers have made things much harder then they should have been.
• Stringers warp horribly, so get your decking on quick. (see the first gotcha)
• Galvanized pipe works great as simple foundation stakes. You can cut it with a metal cutting blade, and drill holes in it with normal bits. Then screw it right on to your wood.

There are some pictures in the More Summer 2009 flickr collection.

Easter 2009 Photos Updated

There are a bunch more photos in the Easter 2009 photoset now.  I guess I forgot to upload them.  They include visits to both of our parents for easter egg hunts, some family portraits by Jan, and a picture of all the Kammerath grandkids when John and Heather came to visit for a late Easter. I also decided that some more photos belonged in the Spring 2009 set, and they are there now too.

Dynamic Website Gotchas

List of things I found out trying to create xsiding.com, when I chose to eschew flash and go with fancy DHTML.

1. ie7 has some problem with embedded sound, so use SoundManager2 (a javascript triggered flash player) to do your sounds reliably.
2. Firefox doesn't allow negative z-index settings if you put it on a css class. Instead use an id (#layeredDiv) and you are OK.
3. Firefox doesn't like backgroundPositionX or Y (which is actually correct since it is some IE monstrosity that Safari implements as well). You have to use backgroundPosition and set both x and y.
4. Everyone is ok with .cur files for custom cursors, so don't bother trying png or something else. A .cur file is just an .ico file renamed. Works great with the .ico Photoshop plugin. Actually has better transparency than gif.
5. Conditional ie code using the [if IE] or [if IE 6] thing can be very helpful since css positioning is frequently off on IE. You can improve the situation by turning off quirks mode with a strict doctype.
   
6. If you set the target="_top" on an href with an onclick handler, IE ignores the onclick (IE 6,7,8)

More to come as I hit em... Notice I don't have any complaints about Safari. If anything they seem to really try to be compatible with everything... Great job Apple!

Delete email from postfix queue containing XXX

I don' t know why no one would just put this simple info out on the web.  Here is a brain dead search and delete of the postfix queue.  It is not fast nor cool, but it works.

find /var/spool/postfix | xargs grep -l "your string" | sed -g 's|.*/||' | postsuper -d -

I am such a UNIX scripting wimp.  This should have been so easy...  Maybe that is why it wasn't on the net... ;-)

Some other aliases that answer missing postfix functions (bash):

Delete all emails with sent by the argument email address:

function delfrom() { mailq | grep "$1" | sed -g 's|\ .*$||'| postsuper -d - ;}

How many emails are in the queue going to a given host:

function hostq() { mailq | grep "\@$1" | wc -l ;}

Recovering Photos From an iPhone or iPodTouch For Free

I hunted around till I finally figured this one out.   Apple won't let you mount these new devices as a drive I guess to prevent hacking (which as always only made people hack it).   There are programs out there to do this which cost money, but most are not even compatible with the iPhone and touch yet.   So I had to hunt...

I used a program called "DiskAid" (http://www.digidna.net/diskaid/) to copy the "Photos" directory from my iPod Touch.  This gets you the lovely ithmb files which are basically big image archives.  Then I used a quick and dirty but capable program called "Keith's iPod Photo Reader" (http://keithwiley.com/software/keithsIPodPhotoReader.shtml) to extract the images from there.  

The second program is only for use by the hacker as it doesn't have any idea how to decode the images unless you tell it.  For the iPhone/iTouch though I'll give you the settings you need.   First open the .ithmb files starting with "F3008" one at a time.  Use the setting of 640x480, 16 bit RGB, Flip Endian and 0 byte offset and then extract the files.

Now you've got a big pile of old .pct files which you can convert back to jpg if you want.  

Have fun!

Ok, I think that Merry Christmas Post is old enough

I guess it is time to post again.  I'm moving batesbunch.com to somewhere new.  $200+ to continue hosting with jumpline is way too much.  I'd really like somewhere free since my site is so small now (mostly dynamic).  I'll post more soon.  I have a lot to talk about, just no time to talk ;-).

Merry Christmas!!

I just put our new card up on our flickr album.  Merry Christmas to you all!!  This has been an eventful year, but as always, the changes and challenges have been for the better.   I continue to be so grateful for my wonderful family and extended family!  Heavenly Father has always been so kind to us!  

Hope you all have a very merry Christmas and a wonderful new year!